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FOREWORD 



Educators have often been criticized for viewing tl\eir role merely as dispensers of knowledge 
and for showing a lack of concern with the application of this knowledge to their students' future 
lives. However, the general public has begun to accept thB idea tiiat the educational system has a 
responsibility to assist ali individuals in making orderly transitions to the world of work. Addition- 
ally, during the last decade, a number of innovative activities, proiects, and programs have been 
developed at the federal, state, and local level in which personnel are attempting to link education 
and work. Among these programs and projects are the following examples: career education, 
Experience-Qased Career education (EBCE), Part D exemplary proiecfe and other proiects in voca- 
tional education, 1 itie IV C of the Elementary and Secondary Education Act (ESEA), Fund for the 
Improvement of Post Secondary Education (FIPSE), and Title I and Hi of the Comprehensive Em* 
ployment and Training Act (CETA). 

Personnel associated with education and work programs are becoming increasingly aware of 
the need for information related to accountability and needed improvements. Many reports-some 
informal, some in the literature— indicate a wide and exciting variety of approaches to evaluating 
programs that link education and work. However, the need exists for practitioners to become 
acquainted with evaluation ideas and materials available for particular situations. 

Recognizing these trends, the Education and Work Group of the National Institute of Educa- 
tion (NtE) contracted with the National Center for Research in Vocational Education (NCRVE) to 
develop the Career, Education Measurement Series. The series includes five comprehensive "user 
oriented" handbooks intended to help local education personnel in measurement and evaluation. 
The handbooks in the series are: 

• Assessir.j Experiential Learni'-g in Career Education 

• Career Education Measures. A Compendium of Evaluation Instruments 

• Improving the Accountability of Career Education Programs: Evaluation Guidelines and 
Checklists 

• A Guide for Improving Locally Developed Career Education Measures 

• Using Systematic Observation Techniques in Evaluating Career Education 

This handbook, "Improving the Accountability of Career Education Programs: Evaluation 
Guidelines and Checklists," is designed to help practitioners plan, mplement, and use the results 
from evaluation to make career education programs more accountable- The handbook provides 
checklists for reviewing your own evaluation plans, instruments, and reports. 

The Center is particularly indebted to Dr. Kay A. Adams, Evaluation Specialist, and Dr. Jerry 
p. Walker, Associate Director for Evaluation, who prepared this document. Special recognition 
should be given to Dr- N. L. McCasim, Project Director, for reviewing versions of the manuscript 
during development and for coordinating user trials. Thanks are extended to Michael Neuman for 
his tfxcetlent editorial review. 
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A speciaS not» of thanks is extended to Dr. Ronard Bucknam^ho originally conceived this 
handbook series and continued his invoivwnent throuj^ their devefepmeitt ts Project Officer for 
the fyiational Institute of Education. Valuable advice in the design end scope of the handbook series 
was received from an ad^jiw^ committee composed of Dr. Robert Ebel, Michigan State Univer^ty; 
Dr. Margajiet Ferqueron/State Direct of Career Education In Florida; and Mi Oeede Sharp®, Georaia 
Department of Education. 

In an attempt to make this handbook truly "user oriented," credit is given to those career edu- 
cation practitioners who participated in the user trials prior to publication. Their valuable assistance 
has greatly enhanced the utility of this handbook. These indivkJuals included: hUn. Irma Adair, 
Principal In Arizona; Sister Annene Si^nmorgen, Principal in Arizona; Mrs. Olive W. Thomas, Career 
Education Coordinator in Florida; Mrs. Barbara C. BaitJe, Pr^t Coordinator in Illinois; Mrs. Maxine 
Thompson, Career Education Coordinator in Illinois; Or. Edward Fernandez, Director pf Secondary . 
Curriculum in New Mexicc; Mrs, Kolene F. Granger, School Coumelor in Ut^; Dr. M. tarry Petersen, 
Director of Vocational anc Career Education in Utah; Mr. Raymond A. Lambert, Housemaster in 
Vermont; Mr. Hertert F. Shipman, Director of Career Educ8tt>3n in Vermont; Ms. Ginny Zahimr. 
Career Education Teacher in Vermont; Walter Faulkner, Career Education Consultant in Vermcnc 
Donna *C. Martin, State Director of Career Education in IHinois; Emit R. Mackey, State Director of 
Career Education in Arkansas; and Dr. Stanley Laavitt, Supervisor in Utah. 

Finally, a special note of appreciation is given to Ms. Mariane Linton, Ms. De^; Roof, and Ms. 
Kathy Kush who typed the manuscript of this publication. \ 
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INTRODUCTION 

I 

Gmny Goruales is a ctuneer educstion project diractor in the Cdfitrai City School District Her 
projifct is a community-bAsed effort to introduce students to careers through on-ths-job C4rstr ex- 
ploration. The project it nine month* under way in a twelve month contract The project may be 
funded for another year if it appears to be successhjt, but th« majority of the members on the focat 
board «ire skeptical about the project They are c^ncwned that the project m«y be too expentiw 
and take too much time away fram learning the basics. As a project director, Ginny n«eds some 
convincing information about the impact the project is having on the stuchwits. Ginny believes in 
the project Many of the students have enormous potential and creativity. They are more int«i»t0d 
in learning ami care^ planning than they have ever been before. Ginny believes tl^ e project iti having 
impact. 

But Ginny is also very busy. She simply has not had much time to think about evaluating itm 
project Her priorities have been els^h^e-ijettins the project startled, solving problems, tinkirtg up 
wofk experience for the students, arHl so on. To Ginny, evaluation is something of a frill. Yat at the 
same time, she is very aware that in three months all her effort could be wastedjs Ginny nas had some 
trainii^ evaluatiott^rid research but doesn't feel she has die skills to davelo|ft "first rate" evalua- 
tion^ Ginny also realizes^t even the highest quality evaluation might not be believed by the board 
if it Is conducted by staff l«^8teat to the project because internal staff might be perceived as having 
\»med interests. 

Time is short. ResouVceipre far from plentiful. After jugoting the constraints, Ginny dec idea 
that it would ber^ficial to mre an independent mpmt to do the evaluation. Yet becaui»i of time 
and dollar {jiniltdtions, it becomes impossible to locate someone to do the job. So Ginny cucidds to 
tackle the valuation herself. She has in her ftfe$ two evaluation instrumttots from other cmmr edu- 
cation programs which ask questior^ such as "How do you like the program?" from 1 not at all 
to S » extr-vnely well; "How much do you feel ycHj learrwsd?" from 1 - very little to 5 «• very? much; 
etc. Ginny combines some of the items into one instrument and asks a member of the project staff 
to critique it 

The revised Instrument is administered to students in the twelfth month of the project. The 
results are positive, and Ginny presents tiiem to the school board. But the project is not funded. 
Although ihe results were positive, the board did not really believe them. They had too many ques- 
tions about the program which were not answered through the evaluation. 

The evaluation of GInny's career education project was weak in several respects. For example: 

• Ginny did not s^rt thinking about evaluation until the ninth month of a twelve month 
project. This left little time to plan and use resources wisely. 

• The evaluation did not address questions that the audience for the evaluation-the school 
board-were most interested in finding answers to. 

• The evaluation was viewed as protecting self-interests and not being an objective view 
of the project's impact. 
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• The eviiuation rtsutts w«rt stibjictivi— «tudf nt*' OfjJniom about what th«y f »am«d and 
how t^ey tiked tiw proflram-rsthftf than mfiftcurwn«nt of actusl skiH, knowledge, or 
attitudi'wiUttBmnwiti. 

• No comparattvt informition thrtxiQh dfivicai such m pm-pmt xminQ wm coiiacted to 
cieiermiitw if tha pro^mm had ^sttuaHy chmg$d itycknt's knowledge, skiits, or attitudes 
about caraanK. 

• The potantidt r^lity of th'4 lK>ttrds' i>a9at«ve attitude toward f acttitating caraor exploratbn , 
for disadvantagad students '^vc* not comtcierad m dasigning tire avaluation. 

• The proiac: diractor who raportod the results to the board vm parcawad as biased, so tho 
board miatrustad the rasuits avan mora. 

r 

This scenario illu^tratfts soma of the ways tnadequata avaSuations cwn causa projeos like Ghiny's 
to fail. As an administrator of career education, vou may be asked to justify the use of funds for 
proflrsms under your iurisdictlon. To iastity your program, you need "credible" evaluativ© informa- 
Xi(m about the effectiveness of the pit>dram. To cdstain this information, you may ^ploy the services 
of a specialist in evaluation research, use someone int^nal to your project staff, or conduct the evalu- 
ation yourself. 'iVhatever the mode, the results of your evaluations, iike Ginny's, may ba considered 
inadequate for one or more of the folkmring reasons. 

• The evaluation is viewed as being off-base because: 

— the wrong questions v«-ere asked 

~ trivial outcomes were m enured 

— tSie evaluation failed to measure the real outcomes 



• The person who conducts the evaluation is viewed as: 

— having v«»sted interests 
lacking evaluation expertise 

— lacking familiarity with career education 

— being an "ivory toVver" researcher who lacks common sense 



• The evaluation results are not believed becausef 

— the data are highly subjective 

— no comparative information is collected to determit j if the program has truly made a 
difference . « 

— tlie evaluation methods are weak (e.g., faulty research design, biased sampling, invalid 
instruments, or inappropriate data analysis) 




• Th« svalyatlon rmiitt are not miHd i^^s^mi 

— th«Y are cc»mmiintcat»d pooriy or not at ail 
_ they sre not believed 

— thtty aren't u<etu! 

— they ctajkh with poiittcjil re«iiities 

— they are wordy and highly technicat 



OrganijEattoii of This Handbook 

This handbook is divided into thirteen units. Eac unit is written to ttand stone. The unite 

are organised Into a logicaf sequence from planning and implementins ^ using svaluatic»is. How»iw. 
the handbook is designed to allow you to skim, skip, or study units according to ycHsr interettt, knowl- 
edge level, and needs. You are encouraged to use the handbook flexibly. 



O^ectives 

» 

This handbook is designed to show how improved evaluation can help make career education 
programs like Ginny's tare better in tests of accountability. This handbook focufsi on the adminis- 
trator's roie in improving the quality arKi u sinfulness of career education evaiuaticMms. Specif ionHy, 
administrators will improve their skills in: 

1. defining thttir responsibilities in evaluation activities 

■ 2. selecting and using evaluation services in a cost-effective and mutually satisfactory manner 

3. judging the quality of evaluation designs, instruments, and result 

4. assuring that the evatustion wiM provide information which is beisevebld, reiev&nt, and 
understandable to key audiences 

5. communicating the results of the evaSuation to different audiences {e*g.^ school boards, 
parents, iegisEators, etc.) 

fi. using evaluation rousts for making decisions 



. ... _ .1 1,. . .. .M^ ,.„^ . U 
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This handbook ii deslgrwd for loct} and state dsrecton of pranimt ' link tducation and woHc 

Since these fN-ograms vary In size and fundina tevtl, adminittra^'invofvemant in «valuatfon of liii 

programs also varies. Some administrators manage programs with differentiated ^^n]^«id cart p!an 

to use external evaluation services. Other admlnistraton, like Ginny Gonzaiei, work urKipr isvert 

res^ktions. Regardless of the ^e and funding tevel of your piYjgram, this handbook wil) hoin you , 

in administm-ing the planning, implementation, and use of evaluation studiesL 



Evaluation Prof ile I 
-n— ^ 1 



The handbook is organized aroutid an Bv^uation Profits, The profile summarizes some critical 
ingredients for planning, implenHtntlng, communicifting, and using the rewiss from evaluations of 
career education proi^ms. It highlights evalueticHi issues tiiat require your direction, input, or sanc- 
tion. The profile contains the frmjor questions that adminis^tors might axk but is not a complete 
list' Hopefully, It will trigger additional thous^tt and Issues for you. . ^ 

The profile is designed to help you: 

• think through your role in the evaluation process ; 

• Idmtify potential weak s^ts in your eveiuation plans ' 

• bKome famiUar with evaluatior. Issues you may Viot have considered 

• gain an overview of tiie contents of the handbook 

• decide which sections of the handbook you should study in depth, read casually, or skim 



Instructions for Compietii^ the Evaluation '^roflie 
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in the left harKi column, rate your current competanc^ to fully siiswer etch of the que^ions con- 
^mtng the evaluation of a career education program by darkening the appnH^riatt circle. Unit numben 
in the right hand column provide the key to the "dictionary" marking of the units thrbu^tout^ handbook. 

Example^ 



CURRENT COMPETENCY 



EVALUATION QUESTIONS 



O • O O O 

O O O • O 

None Moderate H{(^ 

0 12 3 4 



1. Purpose . What types of 
evaluation are needed? 

^' Audience. Who should the 
evaluation senm? 




CURRENT COMPETENCY 



EVALUATION QUESTIONS 



O O O O O 1. PurpotB. What tvpw of tvaluition are 

naaded? 



O O O O O 2. Audience. Who ihould tha evsluatton 



O O O O O 3. Queitioni. What spaclf quaations 

thisuid the evaluation aniwar? 



O O O O O 4. Pfocws. How will the evaluation be 

accomptiihed? 

O O O O O 5. Staff. What type of evalustton services, 

if any, should i» used? 



O d O O O 



O O O O 



O O O O O 




6. Resoonsibititv/Authoritv. What are the 
r^pective roles and rsipontibllities of 
program staff and evaluators? 

7. Uniqueneit. What unique features of career 
education influence its evaluation? 



8. Ptanninfl'StWKlards. What are the character- 
istics of a "good" evaluation plan? 



O O O O O 



9. Instrument Standards, What are the charac- 
teristics of a "good" evaluation instrument? 



o o o o o 



^0- Politico. What political factors might 
affect use of the evaluation r^ults? 



O O O O O 11. Report Standards. What are the charac- 
teristics of a "good" evaluation report? 



O 0 O O O 12. Dissemination. What techniques will be 

used to disseminate the findings? 



O O O O O 13. Use. How will the evaluation findings 

J u! u be turned into action? 

None Moderate High 

0 1 2 3 4 
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The remaiFKief of the handbook is divided into units that ara k^'xi to each ftem on th^ prof ile. 
The large numbers m thu rrght harKi smrgins identify all the pages of a unit Thei» units are or^n- 
\zBd around checklists for as^ir^ m more detail the different aspects of ycHir evaluation plan, in- 
struments, and reportsJl \n each unit the checklist items are elaborated throu^^: 

m short explanatory readings 

^ examples to illustrate key concepts 



practice exerc 
gHiidetinas for 
sug^ted read 



ses to help you apply tl e information to your own setting 
mproving your evaluation 
ngs for further information 



Each unit is written to stand alone. You may choose to read only those units which interest 
you the most or about which you have the least knowledge. The following chart provides a brief 
description of the contents of each unit, and its location in the handbook. 



UNIT 
1. Purpose 



2. Audience 



3. Questions 



4. Proces 



5. Staff 



CONTENTS 

This unit discusses the need for evaluation and the 
pros and cons of using different types of evaluation- 
objectives based^ goal free, audience bussed, cost 
effective, descriptive, and different combinations 
of these and other types. 

This unit stresses the importance of working with 
people and designing evaluations around specific 
individuals' information needs. Guidelines for in- 
volving different audiences in the evaluation process 
arc presented. 

This unit provides mini checklists of questions 
about career ^ucation programs and their out- 
comes that could be answered through evaluation. 

This unit will help you manage the evaluation process 
through specifying t^s, scheduling time lines, 
describing products, and identifying critical de- 
cision points. 

This section offers advice on using evaluation 
wrvices within limited resources. 

f continued on next page) 
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33 



37 



47 

53 
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CONTENTS (contmued) 



PAGE 



6. ResponsibHity/ This section wifl help you decide which evalusdon 



8. 



9. 



Authority 



Uniqueness 



Planning 
Standards 



fnstfument 
Standards 



10. Politics 



1 1 . Report 
Standards 



Dissemination 



activities require your feadei^htp, wh;ch ra:juire 
your support, and which should be delegated to 
others for credibility or efficiency. 

This section outHnes some of the unique features 
and problems of career education that should 
be considered when designing evaluations. 

Advice on reviewing evaluation plans for com- 
pleteness, technical adequacy and usefulness 
is offered. 

Advice on reviewing the content, technical 
features and format of evaluation instruments 
is offered. 

Ideas for dealing with the political factors that 
affect use of evaluation results are discussed. 

Advice on reviewing evaluation reports for com- 
pteteness, accuracy, utility, and feasibility is 
offered. 

This section provides guidelines for sharing the 
results of evaluations with different audiences 
through popularized reports, executive sum- 
maries, news releases, presentations and other 
media. 



59 

6& 

69 

73 
77 

79 



85 



13. Use Advice on using evaluation results to bring 

about change is offered. 89 



PLANNING THE EVALUATION 

A i^ood plan is b^c to anv evaluation proce^ An evaluation plan will help you think throu^ 
needs vK^af information wfisn and In \^i^t format. 

A "good" evaluation plan is one that will 

• help you think through the em re evaluation proces 

• provide a workir-s frame of reference for making decisions, handling crises, and checking 
that activities are still on target toward the original goals 

• be carefiiHy refined, continuously updated, and sometimes drastically changed as vou 
learn from experience arej discover new opportunities 

• be a means to an end-a "working" document— rather than an end in it^lf. 



1. Purpose. What type of evaluation is needed? 



An evaluation can serve a variety of purposes. The purpose of an evaluation depends on (1) the 
type of decisions that need to be made as a result of the evaluation, {2} the phase of the career edu- 
cation program, and (3) the infr -mation needs of those who are expected to use the evaluation 

results. 

There are at least four basic types of a^atuation. Each type can help monitor a different phase 
in the life of a career education program. One type of evaluation Is done In the initial planning ph»m 
of a program so that the effort can be focused on meeting the important needs. This type of evalu- 
ation, often referred to as rmds assessment, is used to help make planning decisions, A sacond type 
of evaluation, often called formative evaluation, occurs while a program is being implemented to 
collect information about how it can be improved. Formative evaluation serves "implementing 
decisions" (decisions about how well the program is being implemented as planned and what aspects 
of the program should be modified). A third type of evaluation focuses on the sum worth of a pro- 
gram-its siccess in achieving objectives, overall eff ^tiveness for the funds expended, and impact on 
Intended eudiencu. This type of evaluation, referred to as a summative evaluation, often occurs 
Viear the end of a program but can occur whenever attainments occur, Summative evaluation is used 
to make "recycling decisions" (decisions about the future of a program or part of a program}. A 
fourth type is an evaluation of the evaluation by an extemal party to verify that the evaluation is 
relevant, accurate, and reliable. This type, called an evaluation audit, is usually connected with an 
evaluation of the sum worth of a program. Figure 1 portrays these four types of evaluation. 

The Purpose Checklist below outlines these four major types of evaluation and a variety of 
more specific types. Your evaluation may include one or several of these tyf^s. Use the Purpose 
Checklist and the additional information in this unit to think through the major puipo^ of your 
evaluation and the types of evaluation that are most appropriate for your situation. More detailed 
information about the needs of the audiences that the evaluation should serve and about the sf ecific 
question's that the evaluation should address is provided in units 2 ("Audiences") and 3 ("Questions"). 
The materials in the sub-section following the Purpose Checklist describe each type of evaluation in 
detail. 



TyiMtt of ivi^uat^ 



Phase of a 
Carmr Ed Program 



0€cu»on 



Evaluation 




Ne«d5 

Asse^inant 



Formative 
Evaluation 



Summatlv* 
Evaiuatfon 



Evaluation 
Audit 



r 



\f PURPOSE CHECKLIST 



instructiom: Answer of the following questions for your career education pr03ran\ by 
checking ( / ) the appropriate response in the space provided. 

Yes No 

□ □ 1.1 Need. Is any type of evaluation actually needed? 

□ □ 1.2 Needs Assei^rnent, Is information needed about the priority needs and 

probf&ms the career education program shouicf addre^? 

□ □ 1.3 Formative Evaluation. Is information reeded to monitor and/or improve 

the program as it progresses? 

□ CI 1.4 Summattve Evaluation, is information needed about the $un^ worth of the 

program? 

□ □ 1.4.1 Objectives Based Evaluation. Is information needed about the success 

of the program in meeting its intended outcomes or objectives? 

□ □ 1,4.2 Goal-free Evaluation. Is information needed about how well the 

prc^ram meets standards of excellence for career education pro- 
grams? 

□ O 1 .4,3 Audience Based Evaluation. Are the information needs of various 

audiences to be answered throuon the evaluation? 

□ CJ 1.4,4 Follow-u p Evaluation. Is information needed about the long term 

impact of the program on participants (e.g., career placement, Job 
satisfaction, job competency, etc J? 

□ □ 1.4.5 Cost-effectivenei^ Evaluation. Is information needed about the cost 

of the program in relationship to the outcomes produced? 

□ □ 1.4.6 Comparative Evaiuation, Is information needed about how your ^*o- 

gram compares with other career education programs on commuii 
criteria? 

□ □ 1,4.7 Descriptive Evaluation. Is in-depth descriptive information (case studies, 

systematic observation, inniepth interviews, etc.) needed about the key 
features, activities, and outcomes of the program? 

iZj l3 I.G Evatiiation Audit, Is infunTiatiori tieeued to verify the accuracy uf the 

evaluation? 
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it is possible that ; .valuation is not actually 
needed. Many evatuations are conducted 
causfi* somebody thought ittey were a good 
idea. A tima^onsumfng and costly evaluation 
may be conducted and then never used. As a general rule, an evaluation shCHitd not be conducted 
unless some commitment can be generated for using the results to chan^t; the prcfjramr set priorities, 
influence decisions, or justify fundir^. 

Even evaluations that are mandated : lould be carefully thought through and negotiated to pro- 
vide useful information ratiier than to cc»iiply with requiremeHits. There are many legitimate i^asons 
fot conducting evaluations. These rang^ from substantiating claims that a project is doing well to 
keeping the public informed of the goals and pro§r^ of career education. The important point is 
to determine at the beginning rather than at the end of the evaluation process how the r^lts could 
be used. If you cannot picture the remits being us^ by one or more specific individuals for at l^st 
one specific purpose, don't do the evaluation. 



1.1 Need. ,s any type of evaluation 
actually needed? 



1.2 Needs Asse^ament . Is there a need 
for information atmit the priority needs 
and problems that the career education 
program should address? 



Needs assessment is a proce^ for identifying 
gaps between "what is" and "what should be" 
and for determining priorities among these 
gaps or ^'needs." Needs a^essment is a useful 
tool In die planning stages to focus a program 
in the right direction. The process of conduct- 
ing a needs assessment will not be covered in depth in this handbook. For more information, refer 
to Needs Assessment and Career Education: An Approach for States \McCaslin and Lave, 1976). 
An excellent review of needs-assessment models and procedures for general education is An Analysis 
of Needs Assessment Techniques for Educational Planning at State, Intermediate, and District Levels 
{Witkin, 1975). Many of these techniques can be adapted for career education. Several states have 
developed needs-assessment instruments. Some exemplary instruments are those developed in Mich- 
igan (State Department of Education, 1976), Wisconsin (Gessner, 1376), Florida (Adams, 1977), 
and Texas (Texas Education Agency, 1975). For abstracts of these and other needs-assessment in- 
struments, see Career Education Measures: A Compendium of Evaluation Instruments, another of 
the five handbooks in this CEM series. 



Once you know which needs.are top priority in your locality, you will be more able to design 
programs to meet these needs. Using data based information about needs as a basis for program 
planning, will also allow you to justify your program to sponsors and constituents, A needs assessment 
can be oriented to student needs, program needs, or both. A needs asse^ent typically s^ks input 
into the educational process from a variety of audiences, including students, educators, employers, 
parents, and other community members. 

Basis of Needs Assessment 



Objectives form the basis of an assessment of student needs and are usually stated as desirable 
skills, knowledge, and attitudes for students to possess. For example: 



"Students will understand the variety and complexity of careers in the world of work." 
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Pu rpowt of Studcnt-needi Asscssn^gnt 

The purpose of a student-needs assessment is to find out it\ how weii student are currentty 
performing on each objective, (2) which objectives teachers, parents, administratmis, students, com- 
munity mwnbers, end other relevant persons perceive to be most important for students to perform 
well on. The bigger the difference between these two dimensions (pw^ormance and importance), 
ths greater the need. A partial sample of a student needs assessment imstrument drawn from the 
ffor^da Assessment of Needs in Career Sducetiofi-fA^CB (Adams, 1977) Is illustrated in Figure 2. 



Figure 2 

Partial Sample of Stuc^t-Needs Assessment Instrument 



Actual Att^fumnt 

Rulistically estimate tile percent of ttudents 
in your district or ich(x>l vtiio hive attaint 
minimai compietflncv in each of the fotlowing 
goats by the time they leave hi{^ school. 



(V9% 10-2«% 3&4S% SO-74% 7&89% 90-100% 



Deeired Priority 

HtVB tiie reiative priority of meeting each 

goe! for your district or schoof. 



Work VqIims 



Loww 



1 



Medium 



Highw 
5 



□ n □ n □ □ 

□ □ n □ ' □ o 



Students uncterstand how careers O Q Q Q O 
contritHite to society. 



Students recognize the social wid 
economic befwfits of working and 
unctentand the cons«c|uenees of 

not working. 

Stuc^nts view career roles iraie- 
pendent of sex stereotypes. 

Stuc^nts view career rolei inde- 
pendbnt of racii^ stereotypn. 



□ n □ D □ 

a □ D □ a 

□ D □ □ n 



Purpose of Program-needs Asse»ment 

A needs assessment can also focm on pre^ram needs. Program needs include the resources, 
activities, people, services, ete, that are required to deliver car^ education to students. Some se- 
lected areas and a rating format for assessment of program needs for career education, also drawn 
from FANCE, are listed in Figure 3. A program-needs asMsment could be used to determirte the 
ncc^Mary st^s for liTipiemeTitlrig s csreef-aducatlon pn^grsm, ^^^knt^cs In the current dclivsry 
system, or priority areas for new funding. A needs assessment process similar to that used for stu- 
dent needs can be used to determine program r^eds. Howev^ in the latter proc^, tiie re^ndent 
is asked to rate the "need for improvement" of career-education r^ourci^ community involvement 
work-experience opportunities, staff expertise, ete. This scale combi?^ the comparison of current 
and desired status into one rating scale, in other vv<»xis, respondents are a^ed to compare in one 
step tiie current status of career educatfon with the desired status and to use inforrpation to deter- 
mine the areas that most rwed to be improved. 
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FigurtS 

Partial Sftmpki of a Prop^ara-NMdi Assesment 



fkmi for Improvement 



Infuiion of C&iwr Educition 

1 . Integrate career tducation into vcxritionid education. 

2. tnttgrota c^retr ecfcrnitfoo into gm^ni education uibjacts (e.g., 
language arts, mathemat^, social studits.l 

3. Provide units on career education (e.g., job huntir^i skills) within 
existing cmirses. 

4. Provide ciirser educition sepvate from the school curricufum 
(e.g.. carsf r fair s{)on»ortd by local induttrv). 

5. Coordinata student cancer education experience acr<^ Qradas. 

6. Promote preservtce ts^har training in career education. 



What is the relative need for 
expanding or improving each 
orc^ram activity in your diitript 
or tdiool? 

tower Madium Higher 
1 2 3 4 5 



cacao 
a D □ a □ 

□ a □ a □ 

□ a □ a a 

a a □ a □ 

□ □□an 



Size of the Needs Assessment 

A needs assesment can ise as big or sma\\ as required. §t can be used nationwide, statewide, 
dlstric- wide, school wide, or within one classroom to help match a program to students' unique 
skill levels, Interests, and backgrounds. A needs assessment should be initiated before a program 
begins so that the program can focus on high priority areas of need. However, h can also be con- 
ducted after a program has begun and then used to refine the specific objectives of the program so 
they writ better meet the needs of the intended population. 



1.3 Formative Evaluation, is informa- 
tion needed to monitor and/or improve 
the program as it progresses? 



Once a pre^ram ge^ under way, there will still 
be countless problems and ways that it could 
be improved. Even ^e best laid plans need to 
be reaped and refocu^ when they are put 
into pr^tice. There are many evaluation 
techniques that can be helpful to provide 
information for improving a program. This type of information should help project administrators 
to make decisions about what aspects of the program ne€d to be modified. 

Ei^aluaiHn for improvement can be as simple as scheduling periodic nrwetlngs with the program 
staff to discuss the''' problems and to ask for suggestions for improvement. It can also be more 



14 



systematic and elaborate. However, even the moit t iaborate ttvaluatibn for improvenuint is usually 
based upon scMTie fdt-iy simple questicMu; .. 

Are we stiller target towards the program goals? . ^. 

What are the strengths of the program? What are the best features? What's working well? 
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* What are the weaknesses of the program? What are the most probtem-rtckien areas? What's 
not working? 

* What suggestions, recommendations, and ideas do various participants in program have 
.for building on tfie strengths and reducing the problems? ^ 

• What related opportunities or activities are occurring that the program should incorpwate, 
take advantage of, or team fcom? is tfie program up to date with the "state-of-*e-art" in 
career education? 

Although these questions appear to be simple and suaight-forward, obtaining accurate compre* 
hensivti answers to them Is often difficult and time consuming. For guidance in conducting evalua- 
tions, for improvement, see: 

1 

• Lawson, Tom E. Formative fnstructiomt Product £iraluation: fnstrurmnts and Strategias. 

Englewood Cliffs, NJ. Educational Technology Publications, 1975. 

•Sanders, James R. and Cunningham, Donald J. Techniques and Procedure for Fornwtive 
Evaluation. Paper No. 2. Portl^d, Oregon. Northwest Regional Education Laboratory, 
1974. 



1.4 Summative Evaluation, is informa- 
tion about the sum worth af the program 
needed? 



Judging merit or worth of a program is the 
heart of an evaluation. There are many dif- 
ferent ways to assess outcomes, impact, merit, 
or worth. For example, you may decide to 
focus the evaluation on the intended outromes 
or objectives of e program. Your question . 
might be "How well were the Intended. outcomes achieved?" However, at some point in the evalua- 
tion procesSA^ou may realize that the objective do not do a very good job of dexribing what really 
happened in Vie program. The objectives are often out of date; perhaps the program has matured 
and changed sirice they were written. 

This discovery may bring you to a second evaluation question: "To what extent have the pro- 
gram objectives been implemented as planned?" Sometimes there is an evaluation of activities and 
entities that nsv&r really exlst^; W?^«t f 1974) provides tfie following example: "The concept 
of team teaching has tared poorly in several evaluations, resulting in a general impression that team 
teaching is ineffective. Closer Inspection shows that many methods labeled as team teaching pro^ 
vided no real opportunities for staff to plan together or work together in direct instruction." It is 
often valuable to find out wrfitch parts of a program have actually been implemented according to 
the original plans. 
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Usas of Summattvo Evatuarions 

It is usustty difficult to picture the impact of a program if onSy the i ntendfid outcomes are 
evaluated. This is not to say that intended outcomei should be ignored. Many tirms ob|ectives, 
as stated in a pro|ect p70p&m\ or otiier plannir^ document, reflect a commitment to your sponsors 
and constituents to work tenvard certain types of CHiti^imas. However, you can also make a case for 
evaiuatins more than your intended outcomes. As a program is implemented, it matures, grows, and 
changes. Some things work and o^ers don't. P'^ogram staff who are learning from faitunN ami suc- 
cesses are modifying tfte program as it progrmses. 

♦ 

Therefore, in addition to evaluating the attainment of intens^ outcomes, the evaluation can 
also be u^ to: 

* descHN key features, activities, and outcomes of the program in dep^ 

' * determine tht; unintended positive outcome of the program, such as improved attitude 
toward school, reduced at^enteeism, etc. 

I 

* determine ^y unintended negative outcomes of the program, such as declining achievement 
xores^ increased student disruption in non career oHented school subjects 

* determine how well the program meets standards of excellence 

* determirite the impact of the program on desirable short-term student outcomes such as 
knowledge of careers, knowledge of self in relation to careers, job-seeking ^tlis, etc. 

* determine the impact of the program on desirable long-range sttli^ent outcomes such as 
job placement, job competence, and job satisfaction 

* determine if the cost of the outcomes produced is comparable to the cost of programs 
with similar outcomes *. \ 



1.4.1 Objectives Based Evaluation, is 
information needed about the success of 
the program in meeting its intended out- 
comes or objectives? 



Objectives provide an important and solid 
base for evaluation. Objectives-ba^ evalu- 
ation is the most common type of evaluation 
practicKi in education. The various terms used 
to describe objectives can become ^mewhat 
confusing. Goats, intended outcome;., expected 
outcomes, objectives, behavioral objectives, per- 
formance objectives, elements, and measurable objectives are some frequently used terms. Basically, 
goals are more general statements, and objectives are more spacifk:. They both describe intended 
or expected outcomes of a^program or learning experience. Goals are usually stated too broadly or 
generally to measure direqtly how well they have been attained. Therefore, evaluations often rely on 
more ^^ecific objectives ^at are derived from goals. An objective is said to be statedjn "behavioral," 
"performance/* or "measurable" terms when It clearly describes what a person should be able to do 
after completing a learsling experience. A complete behavioral objective can contain four parts {Adams, 
Lowry, and Suzuki, 1977). The terms describing these parts begin with the first four letters of the 
alphabet, "ABCD." 
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6. Behavior, A imowlttdge, tfcMi, or attitude that the actor can b« fs«n or hear^ 

J / 

C. Conditiof} (an optional part of behavwal obiectivei). What t*w actor Hm to work with 
to (^onn the desired bdiavior; it can tm materiais or a settins. 

D. Oegrm of Success. A cntorion to determine vvhether.an actor has succesttuHy mastered 
the i)eh4^ic»- prescribed by the objective. 

Objectives based evaluation depends on five basic steps: ; , - ^ 

1 . Establish and validate* broad goals for the program. 

2. IdwitJfy and validate specif ic objectives that are derived from each soal> Theae objective 
may desf^-ibe knowledge, skills, attitudes, or behaviors. 

3. State each objective in measurable terms. ^ 

4. Develop or select measures of performance for eacMlolective. 

y' 

5. Measure performance on each objective. Performance can be measured bifore and after 
the program thr*' -^post testing to^termlne the amount of chariQe in participant!' 
knowledge, .^iils, . ^m, or beh^vibrs. Or performance can be measured only at ttm end 
of the progriam to assess how^^H^tarticipants have mastered program objectivei. 

You mvS members of your ^ff should xsx^t respon»bility fsK* st^ t and 2. St^ 3 should be 
a joint activity between progmm staff and evaluators. The progmm evatuator should take responsi- 
billty for tfie remaining step^ If you can clearly communicate your intentions for the progrwn, tlw 
evaiuator should be ableio translate your intuitions into behavioral objectives. You will, of courw, 
want to review the^l^Rtrabie objectives (ard later, die testsi to eniKire that the or ifiinat intimttons 
nave not been^ilstorted. 

W|)eh developing program goals and objective for career education programs, you ^ouid review 
eKijStkig sets of objectives. Some sources of goats and objectives for career education programs art: 

*Perspectives on the Problem of Evaluation in Career Education-"lm Learner Outcomes" 
{Hoyt, 1976). 

*DevelopmentaI Program Goafs for the Comprehensive Career Education Model {The Center 
for Vocational Education, 1972). 

^Career Education; An Introduction {The Career Education Center, Florida State Univ«iity, 
1975). 



•Validation Is the process of involving program staff, participants, and/or constituents in assess- 
ing the extent to which goals or objectives for the program truly reflect the program's contents. Goals 
and/or objectives are reviewed in terms of their relevance, comprehens!vene», clarity, and feasibility. 
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*C9f^r O6v^apm0nt 6oais amf Parformartce fndicstors (Michigan O^rtment of Education, 
1974). 

*Basi'c LvamBi- Outcomes for C^mr fcfc. stion (Texas Educatkin Ag«iwv, 197^). A tampls 
Ustlng of student ctreer education outcomes is also iisted on pa^es 39 and 40 of this 
4ian)ciboQk. 



L Basing evalbations totsUy cn performance 

1.4.2 Goal freg Svaluntio n. Is iofonma- objectives restricts th« focui of th« evalu^ition 

tio^ needed about how weH the program to intended outcomes. Unintended ou^omes, 

meets standards of excellence and desirable the IntHnsic qualities of a program, and tt« 

outcomes for career education? ratationships among tbe objecttves-rthrae focal 

' points that are potentially as important as the 

wiginal objectives-^are often ov«1odked. 

Goal-free evaluatfcMi was introduced by Scriven (1972). The ratii^le for goal^free evaluation 
can be summarized briefly as follows. First, educational goals should not be taken as gi^; they, 
like anything else, should be evaluated. Second, goals are ^nerally little more than rhetoric which 
seldom reveal the real ob|ectiv« of the program. Third, many important outcomes of a program or 
side effects do not fall in the category of goals and objectives; anyway. For example, a career educa- 
tion prog^^ may sdn;e social goals, such as reducing student absenteeism and tardiness or facilitating 
parental involvement in school ectivities-desirabie outjximes but usually not explksit goaltof career 
education programs. Negative side affects such as declining achievement-test scores or increasing 
student disruption In non-career oriented courses, might alsct result from a career-odiK»tion program. 
Fourth, there are many intrinsic standards of excellence that virtually all csrswr-educction programs 
should try to meet, such as addrwsing important needs, limiting expenses, and so on. 

Objectives based and goal-free evaluation can »ippl^ent one ano^er very well. Most program 
directors and sponsors need to know how v^ll the program is meeting its goals. But goals also serve 
as. blinders, causing one to miss ^ose important aspects of the progr^ that are not directly related 
to goals. Goal-free evaluation ailovtrs the total program to be evaluated on the ba^s of ail its outcoririe^ 
whether Intended or not. C 



1.4.3 Audience Based Evaluation. Are 
tiie information needs of various audiences 
to be answered through the evaluation? 



Stenner (1972} propostd a strategy called 
Information Based Evsluation far Career Edu- 
cation. Stenner recommends that the informs- 
tf on needs of the users of the evaluation ajp- 
plement program objectives as reference points 
for designing the evaluation. Information needs of users are qMestions and concerns about which 
relevant individuals desire answers. 

Potential information users {uid\ as those identified in Unit 2, "Audiences") can be polled fo 
gain representative views of what questions the evaluation should answer. Various types of audiences 
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can be asked to "fill in the blank" for a s^tement such as "( would like to know 

about this career education program/' A balanced list of evaluation questjons that reflects the in- 
formation needs of different audiences should be developed. Since program developers and staff 
generally have a high commitment to tfie program, their information nasds may show them to be 
somewhat myopic and reluctant to receive negative feedback, individuals who are more removed 
from the program may want answers to "hard nose" questions about its imp^t, and they may be 
nwre likely to question potential weaknesses and unintended outcome. 



Uses of an Audience Based Evaluation Worksheet 

After a reasonably complete i.st of information needs has been developed, you may want to 
involve representatives from the different audience in selecting the most important questions for 
the evaluation to answer. A thrt^e-step process is suggested. First, audiences can be asked to rate 
the utility of information as essential, important, ma^oinai, or questionable. Second, selected audiences 
(project staff, project director, and project sponsor) can rate the practicality (time, cost, method, eta) 
of collecting the information as very high, high, moderate, or questionable. Third, both the utility and 
practicality of the information can be used to rank ar^s of information from highest lo lowest priority. 

A sampJe worksheet for recording information needs, audiences, utility weightings, and practical- 
ity weightings is illustrated in Figure 4. The utility weightings can t te tabulated across audiences to 
deternime which fnformation is most important to those audiences. Practicality weightings can 
be used to temper the utility weightings. Some types of information may be very useful but also 
very expensive and time-consuming to collect (such as longitudinal follow-up studies of graduates.) 
Both practicality and utility should be considered when selecting the information needs which the 
evaluation will serve. 



Benefits of Audience Involvement 



The involvement of audiences in focusing an evaluation has an additional benefit. Research on 
educational change has found that when individuals are involved in the planning and implementation 
of an activity, and when they are kept informed of progress and periodically asked for advice, they 
will tend to be more supportive and to mak.^ greater use of the outcomes and products emerging 
from the activity. Therefore, if relevant audiences are encouraged to express their information needs, 
they will be more likely to support and use tlie evaluation results. 



1.4.4 Follow-up Evaluation. Is informa- 
tion needed about the long-term impact 
of the program on participants? 



mobility, etc. of graduates. A follow-up study can 
graduates as well as from employers of graduates. 

Purpose of a Follow-up Study 



Follow-up studies are designed to evaluate the 
product of career education progr ns-tne 
graduate's progress. The primary at of such 
education, the preparation of individuals for 
careers, can best be assessed by examining the 
career placement, performance, satisfaction, 
be used to collect information directly from 



Former students can often provide valuable information about the strengths and weaknesses of 
a career education program. A student follow-up study can serve a variety of purposes, for example: 
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Instructions: 1 . In each cei! on the worksheet, enter the svereae utility rattnfi given to each infonmetion need by 

group based on the following scale: Utility: 4 » essential 3 » important 2 * marginal 1 * questionable. 

2. IntiiefirstTOTALScolumn, enter the average utility rating across groups. 

3. in the second TOTALS column, rate the practicality of collecting the information based on the following t^te: 
Practicality: 4 « very high 3 « high 2* moderate 1 « low. 

4. In the fast column rank the overall priority of the iitformation needs from greatest to least 



t . To dtttwTTtine the atrtm pattt^ns of formm- participants in various career-education pro- 
^vm, Infomriation about the path of advancement and pattern of salary increase for 
fornier students can be tracked a^ intervals of one, three, and five years. Graduates' in- 
voivement In conttnuins or recurrent education can also be assessed 

2. To determine the mobility of program graduates. Movmient from job to |ob, from one 
iocality to another,^ or even from d^artm^t to department within a firm can be assessed 
by a follow-up study. 

3^ To determine the adequacy of the career-education program in preparing students for the 
transition from school to work. Graduates' accesses and dlfficultiis in job hunting, job 
interviewing, job applications, work retationships, work habits and attitudes, job pi^iem- 
solving, etc. can be assessed thrcHJQh a foliowHjp study. . 

4. To determine the appropriaten^ of students' career choice, job satisfaction, personal eiid 
social ^If illment, and av/areness of means ami implfcaticms for &*)anging career choice, 
job situation, and life style can be assessed. Additionally, graduates' future pl«is and 
ability to dianp and ^ance in their career can be determirted. 

Method for a Follow-up Study 

Some step-by-step guidelines for cor^iucting follow-up studi^ are p^id»i in chapters 4 and 
5, "The Follow-up as an Evaluative Tool" of Evaluating Occupatidml Education and Trmning Pro- 
grams (Wentling and Lawson, 1S76). The steps covered include: 

1. Development of the follow-up instrum^it 

— making the instrument visually appealing and easy to resd 

— developing items of different types (yes-no response, multiple choice, rating 
items, open-ended Items) 

— writing directions * 

— devising the format for the instrument 

— validating the Instrument 



2. Administration the follow-up study 

— determining groups to follow up 

— choosing the method (personal Interview, telephone interview, or mail airvey) 

— locating former students 

— orienting respondents before they receive the survey 
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— distributing instruments 

— foHowii^ up non-ra^ndents 

3. Processing th» foilow-up results 
~ tabulating responses 

~ summarizing responses 

4. Utilizing results 

— altering objectives, con^t, and activities of tiie career education prc^ram 

— determining the long-range impact of the career education prog|E»n 

Employers, too, can provide valuable information about former students' job performance, work 
habits and attitudes, and work relationships. A simple instrument like following one can pro- 
vide ^plovers' perc^tions about tiie job traits of ^rmer students. 





Low 


Average 


High 


Oependabiiity. promp^ess, reliability in attendam;e 


1 


2 3 4 


5 


R^onsibiiity. willingness to accept and perform work 


1 


2 3 4 


5 


Initiative, ability to plan and direct own work 


1 


2 3 4 


5 



Employers can also rate the specific career compstencf^ of graduates. 



1.4.5 Cost-effectiveness Evaluation. 
Is information needed about the cost of 
a program in t»'ms of the outcomes 
product? 



The importance of determining the cost- 
effectivene^ of a program is explained in the 
following quotation: 

During the past few years, an increas- 
ing numb^ of schi^l borr^s and levies 
have been disapproved. The once ade- 
quate plea of "we med more money so that we can provide children with a high quality edu- 
cetidn" seems no longer to satisfy the American vo^. School districts are being for(^ to 
indicate more clearly how their present funds are being s^nt, what the result are, and how 
additional funds will be spent if they are approved by local citizens. Additionally, schools 
are f^ed with escalating costs due to inflationary trends. They, too, must pay higher costs 
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for supplies, equipmen^tillties, and penonnel. Tf^s& and other trends are forcing 
school administrators to examine their iHidgets more cicely and to identify areas »tiere 
cost-saving practices can be put into effect (McCasiin, 1977). 

The outcomes of cost-effectiveness analysis shouid tj&^o operate career education programs 
that produce the best possible results for the least possible expen^ A comprehensive evaluation of 
8 program's effectiveness as compare! to its costs can be a complicated undertaking that requiitis 
^eciaiized expertise. Additionally, a cost-effectiveness analysis can lead to an over^TiphfSis cn eco- 
nomic effici^icy. Many of the goals and c^jectives of a career education <^re socially desirable— will 
lead to a better quality of life and work for students-but are not necessarify economically effi^:lent 
This section will not delve into ttm highly technical aspects of cost efff stiveness analysis. Rather, 
highlights of several key topics related to costs and benefits are presented. 



Types of Costs 

When collecting information for a cost-effectiveness evaluation' of a program, you shoAJld collect 
different types or levels of cost information. Th^ shouid include: 

• Total program cost (by year) 

• Per pupil costs |by year} 

• Initial start-up costs (cos^ for materials, getting the program establish^, developing ma- 
terials, estoibli^ing relationships, etc.) 

• Program operation costs (costs after development hs» been completed and the program is 
in operation) 

• Estimated cost for another schcKsl district, etc. to replicate the program. 

• Costs (by year) for major elements of the program, such as: 

— Instruction 

— Administration 

~ Facriities, equipment, and supplies 

— indirect costs or overhead 

— Other (travel, consultants, etc.) 
Cost-Saving Prscticcs 

It is a good idea to maintain records of all the cost-saving practices used during the development 
and operation of the program, for example: 

• Using resource contributions from the community (money, time, facilities, equipment, 
etc.) 
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• Buiiditis on the work of another program or projact which %»md devsiopment tirrui 

• Usrng voiuntsers 

• Mamtaintng cooperative effom with othEH- schools or community organizations in vtHiich 
'costs ar9 shared 

• Using practi«^ that conserve materiais, supplies, duplicating expenses, etc. 

• Using practice that reduce administrative costs 

• Using practices that consorve energy 

The amount of money saved by using th^ practice should also be estimated and recorded. Cost 
saving information may be very persuasive evaluation data when combined with infonmatlon about 
program outconifes. 



Ctost Comparison 

The cost of your progr^^ should be compared to similar care^-education programs in other t 
schools in the district. Since cost connparison can be dec^tive e^secialfy when done on a cross- 
distrtet basis, it is important to ensure liiat the s»ne program el^^ts are being compared when 
costs are ctetermined. Therefore, It is ^ssntial to compare progums m the basis of the different 
types and levels of costs described above (e.g., per pupil cost, Insmictlional costs, %tc.) rather than to 
compare only total program costs. 



Zero-3a^ Budgeting ) 

Zefo4}ased bud^ting is a mana^ment tool to help put money where the priorities are. industry 
t^es tim technique to r^Bssess operations annually from the ground up, wftii every dollar si:mnt justi- / 
f ied in terms of current organizational goals. Zerp-based budgeting should be used (kinr^ the budget- \ 
ing process: 

« 

• to reassess priorities 

• to eliminate duplication 

• to determine low levels of performance 

Chuang 0977) outlines twelve steps for doing zero-based budgeting: 

1. Define educational needs 

2. Formulate goals and objectives, and rank them in order of priority or importance 

3. Define the method for asse^ing e xrh objective 

4. identify overall constraints and requirements 
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5. EstfOiltsh cfiteris for selecting among alternative programs 

6. Develop decision packages (documents that describe alternative ways to achieve objective 
in such a mann^* that ti ley cen be ranked and evaluated again|t all other activities com- 
peting for limited resources ^ 

7. Defir^ the cost of alternative packages f 

8. Define the antieipa^ benefits for each package 

9. Rank the decision packa^ and »lect the most desirable one 

10. f^^pare an implementation plan 

1 1 . Manage the allocation of resources to achieve objectives y 

12. Assess results 



. Single program, noncomf^rative evaluations 

1 .4.6 Comparative Evaluation. Is are the most common tyf^ of evaluations in 

information needed about how your education today, because tiie primary basis 

program compare with other career- for judging »ia»BS ties not in comparir^ the 

education programs on common criteria? program with any other but in determining 

— ^ — ^ 1 ( 1 \ how well tf»e pros^am does what It pur- 
ports to do, (2) how well the program meets 

external standards, and (3) how well the program chan^ students' performance, skills, or attitudes. 

Comparative evaluations, on the other hand, are those in which two or more programs or methods 

are compared with one another on common criteria, and these evaluations can provide much valuable 

information. 



Types of Comparative Evaluations 

Among the several types of comparative evaluation are these: 

• comparing the same program in different settings or with different types of learners (for 
example: a junior high career-awareness program is tried in an uriian school and a subur- 
ban school) 

• comparing a program with a control group's program on the same outcome (for 
exampls: In the same sshool one »nior class studies a unit on job hunting skil^js while 
another senior class studies a unit on entrepreneursliip. Both groups of students are 
evaluated in terms of knowledge of procedures for applying for a job, job Interview skills, 
and ability to prepare a resume.) 

• comparing two or more programs that aim for the same outcome but use different methods 
(for example- Three programs are trying to increase students' knowledge of job categories. 
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major duties, and supply and demand. Om progrwti it using sh(Mt-term community-based 
work expenencos, one program is using a self-study course in .he Khoof iibrary, and one 
program is using classroom instruction with community resource pecHile). 

* comparing the strengths and wi»kne»e$ of various programs setting ^ the mne stan- 
dards (for example, ail t)^ high sdiool careorH9ducation programs in Florida are compared 
as to the extent to which they are meeting the t^ learner outcomes outlined the Office 
of Career Education) 



> 



Character istk^s 



A Sample Comparative Evaluation 

Comparative evaluations )snd to be mom useful when ^ey compare specific characteristics and 
outcomes of different progrwns than when they ccHnpare the relative strei^s ^d woeknesses of 
total programs. For example, consider that several junior high c^w-awareness programs are being 
compared in terms of the foilowir^ characteristics and outcomes: 

Cost 

instructional materials 
Instructional proc^ 
Community invol\%ment 
Number of smdents served 
Length of program 
Setting 

Target population 

Knowledge of job categories 

Knowledge of worker qualifications for 
specific jobs 

Knowledge of major duti^ for ^ecif ic 
jobs 

Knowledge of ^pply and demand for 
different o^upatlons 

Knowledge of job characteristics (hours, 
pay, etc.) 

Attitude toward learning 
Desire to work 

Ability to locate and use sources of 
career information 



Outcomes 



35 



26 



A comparative evaluation of the two or more programs might rev^eaf the following. Program A is 
less eKpensive, has more community involvement, uses an open, unstructured instructional process, 
and involves fewer students (primarily disadvanta^ minorities), it has helpecf students more in 
attitude toward leaming and in desire to work. The weakest outcomes of the program were ability 
to locate and use career information, knowledge of supply and demand, and knowledge of worker 
qual if (Options. 

Program B on the other hand, is more expensive, serves more students ^primarily wburban 
whites), has structured lesson plans with corresponding instructional materials for students, and has 
less community involvement* It has improved students' ability to locate and use sources of career 
information and their knowledge of supply and demand, job categories, worker qualifications, and 
major duties. It has had no measurable impact on attitudes toward learning or on desire to work. 

In comparing these two programs, one finds that Pfiogram A appears to produce primarily atti- 
tudinal changes in a few students with limited resources through various community experiences. 
Program B appears to produce increased knowledge of careers and of procedures for obtaining ca^ 
reer infoi^mation; the cost is greater, but so is the number of students. 



Strengths of Comparative Evaluation 

Comparative evaluations have the distinct advantage of offering greater scientific validity to the 
evaluation process. Comparative evaluations can increase both the internal validity (or accur^y) and 
the external validity (or "general izability") of evaluation results. For an outline of 16 experimental 
and quasi-experimental designs that can be used for comparative evaluations, see Campbell and Stanley 
0966). 



Limitations of Comparative Evaluation 

The primary limitations of comparative evaluations are resources and feasibility. Comparative 
evaluations are typically more expensive and time consuming to conduct. Additionally, testing 
conditions required for some types of comparative evaluations— such as random selection, random 
assignment, and control groups— are often difficult to arrange in school settings. 



■ I A descriptive evaluation provides in^epth 
1.4.7 Descriptive Evaluations. Is in- information about how a program has evolved, 
depth descriptive information rreeded what day-to-day activities have occurred, and 
about the key features, activities, and what the experience has been tike for selected 
outcomes of the program? program participants and staff. A descriptive 
evaluation makes use of tools, such as system- 
atic observation, in-depth interviews, case 
studies and records (e.g., diaries, critical incident records, school records, newspaper accounts, daily 
logs). Two Career Education Measurement Handbooks, Using Systematic Observation Techniques 
in Bmfuating Career Education {Kester, 1977) and Assessing Experiential Learning in Career Education 
^fy^alak, ed. 1977} (other handbooks in the current CEM series) provide detailed information about 
how to evaluate programs using descriptive techniques such as these. 
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Kinds of Descriptive Evatuatton 



A descriptive evafuation often cuiminates in an evaluation report, written in a narrative "story- 
book" style, that describes in detail the experiences of a small sample of program participants. Inter- 
esting anecdotes and success stories about the e^cperiences of these participants are shared. Such a 
report might also present the program from the viewpoint of several of the program staff and com- 
munity resource persons. A description of program aciivities might take the reader through a day's 
experience as a stud&f^t in the program. For an excellent example of a descriptive'evaluetion report, 
see An Evaluation of TCITY, the Twin City Institute for Talented Youth (1971) by Robert Stake and 
Craig Gjerde. 

Another type of descriptive evaluation is the technique called "responsive evaluation/' Respon- 
sive evaluation focuses on activities and describes the experiences and outcomes for a small random 
sample of the individuals involved In a program. A responsive evaluation report would give the reader 
the vicarious experience of being in the program. In Evaluating the Arts in Education: A Responsive 
Approach (1976), Robert Stake, who developed responsive evaluation, outlines reasons for choosing 
the technique and demonstrates how it can be used. 

A third descriptive technique- the evaluation techniques used by art critics, wine connoisseurs, 
gourmet cooks, and other connoisseurs aruj critics of the fine arts— has been applied to education by 
Eliot Eisner. His paper, "The Perceptive Eye: Toward Reformation of Educational Evaluation" 
(1975), presents a strong case for an artistic model of evaluation that should be used in conjunction 
with a more scientific model. 

Because unsystematic descriptive evatuations are often more subjective and less scientificanY 
rigorous than the more traditional types of evaluation, they should usually- be used in combination 
with a scientifically based evaluation. However, descriptive evaluations do tend to be more interest- 
ing than statistical evaluations; they often communicate the essence and life of a prograip more com- 
peiltngly, especially to those not sophisticated in research and statistics. For school boards, legislators, 
and parents, descriptive reports may be especially useful in communicating the activities, strengths, 
weaknesses, and outcomes of a career-education program. Even for audiences with more expertise 
in evafuation methods, descriptive evaluations can be refreshing. Consult the Career Education Mea^jre- 
merit Handbook on "Using Systematic Observation Techniques in Evaluating Career Education*' for 
more specific ideas. 



Definition of an Evaluation Audit 

An evaluation audit is an evaluation of an evaluation. Cook { 1374) describes an evaluation audit 
as an external review procedure performed by qualified outside technical personnel who are not di- 
rectly involved in the actual operation or evaluation of the program. It is designed to (1) as e» the 
appropriateness of the evaluation procedures employed, and (2) verify the results of the evaluation. 



1,5 Evaluation Audit, is information 
needed to verity the accuracy of an 
evaluation? 



The word ''audit" is connected in most of our 
minds with the audit of an income tax return 
or the financial auditing associated with ac^ 
countants, but evaluation auditing does not 
focus upon financial activities. 
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Audit Functlont k 

Tbesa two functions of an •vaiuation tuditor couid be de«:rib«d in graatfirt- datait ss foltows: 
1. Chack and report on ths strei^ths and w^knasses of tiie avaluatioo ptan, inctuding 

- compilsteness of tfw plan 

- validity and reliability of data^oHection instrunients for the career-education pro- 
grann being evaluated 

- internal and external validitv or the data collection procedures and conditicKis 

- technicai quality and ^propriatenes; of the data storage and analysis techniques 

- cost-effacttvanass of tfie evaluation plan 



2. Verify the amiracy of ths evaluation findings and report. The priniary requirements for en 
effective evaluation auditor is indepsndetKe. Fre^mably, an external party can determir^ 
the "truth" without being subjected to influences of various kinds. Cradibility of educa- 
tional results from the program is enhanced if an outside invsistigator with ob/actmty is 
involved in examining program operations and results. 



Audit Process 

The auditor should develop a contract and a plan for auditing. The contract states rates of pay, 
schedules for visitation, dates of report submission, and other specified items. The audit plan d^ribes 
how the auditor plans to carry out the audit, it identifies what c^|ac|ives will be examined, whit in- 
struments examined, what data analyzed, and so on. The auditor typically has the responsibility of 
pr^aring three major audit reports: 

Pre-audit critique— a review of the program evaluation plan to determine if it can be audited 
and to suggest changes in objectives, instrument, data collection procedures, and so on. 

Interim audit report- an audit of the mid-year evaluation report of the program. 

Pinal audit report— an analysis of a sample of the program evaluation data and an audit of 
the final evaluation report of the program. • 

A kit of workshop materials to train educational auditors has been developed (Klein, 1971) and it 
describes the audit prodess in more detail. 



Audit Criteria 

Many criteria have been suggested for the evaluation of evaluations, sometimes called "meta- 
evdluation" CScriven, t969K Many of thc^ are derived from accepted criteria for research, but re- 
search criteria (e.g., fxhnically sound information) comprise only p<<it of the standards that 
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evaluations stiouid meet Evaluations must also produce f tndini^ that are use^l to particular audierKes. 
Additionally, the findings must be worth more to the audierK;^ than the cost of obtaining the informa- 
tion (i.e,, the evaluations must bQcost^ffecava). A variety of specific criteria to use in evaluating an 
evaluation plan are described in unit 9 of this handbook. Stuff iebeam ( 1974) suggests the 1 1 specific 
criterii^ for "meta-eveluations" listed below. < 



Table 1 

Criteria for Meta— Evaluations 



Criteria for Technical Adequacy 

1. internal validity - whether thi^ findings are true 

2. External validity - whether the information is "generalizable*' (i.e., the range of persons 
and conditions to which the findings can be applied) 

3. Reliabitity — whether the data are accurate 

4. Objectiv it y — whether the data are likely to be interpreted similarly by different 
competent judges 



Criteria for Utility 

B, Relevance — whether the findings relate to the purposes of the program 

6. Importance — whether the evaluation covers the most essential feature of the program 

7. Scope - whether the evaluation addresses all of the important questions 

8. Credibility — whether the audience trusts the evaluators and supposes them to be free of 
bias in conducting the evaluation 

9. Timeliness — whether the evaluation findings are available m time to be used in making 
decisions 

10. Pervasiveness - whether the findings are disseminated to all intended audiences 



Criterion for Cost-Effectiveness 

1 1 . Cost-effectiveness — whether the evaluation costs ere kept as low as possibt^without 
sacrificing quality 



In 
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SummarY 



There are four basic tvpes of evaluation: needs assessment, formative evaluation, summative 
evaluation, and evaluation auditing. A needs assessm^t can examine student needs, program needs, 
or botli. Formative evaluation is used to monitor a prosram's progress toward its yoafs and to pro* 
vide periodic feedback for improving ^e program. Summatiwi evaluation^-wfiich may take the fc^ 
of objectives based, goal-free, audience based, foVow-up, covt-effectiva, comparative, or descriptive- 
gauges the total wrorth of a program. Most evaluation ctesigni will use a combinattcm of several type^ 
of evaluation. For example, the evaluation may be primarily objectives based with several questions 
about improving the program, several goat-free qu^*ons, md several descriptive qu^ions. Evalua- ' 
ticN) auditing is an efahiation of ^ evakiatton. 

AttilKMjgh it is helpful to broad^i hf^'izons abcHit the variety of ways to evaluate a program, it 
would be virtually impossible to «nploy thetn all at or^ time. You will have to decide what is most 
important and praaical to evaluate about yoUr prop'em. Determinir^ the purpose of the evaluation 
is a critical step in tH^ evaluation proce^. 
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2. Aodjgficc. Who should tht evahiatidn ssrve? 



Evaluations are conducted to provide umful information for inf iuen&mg ddcisions. Since 
people make decisions, evaluations will be more useful if they are designed witii specif^ people in 
mind. There are usually ^ree types of audience for en evatuation study: 

1 . Decision makers arKi opinion leaders who wiSi um tfte evaluation results to influence 
decisions about the future of tf^ program 

2. Program staff afKj participant who are directly affected by ^e evaluation 

3. The general fmbiiQ who would be interested in hiring about resuSts but ars not directly 
involved in tiie program 

The following checklist enumerates varitnis audiences in these three categories who could po- 
tentially be served by your evaluation. Use the checklist to help decide which ^*diefKes shcNuSd be 
Involved. 
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yf AUDfENCE CHECKLIST 



instructions: In the left-hand column, check whether tha evaluation should serve each of the fol- 
lowing audience In th<i right-hand column, pl;ice the audience checked "y^" in 
rw\k onier from greater to lesser importance . Use "5" as the highest rank. 

2.1 Dacision Makers and Opinion Leaders . Have the sanction and support of reltvant decijion ^ 
makers and opinion laadsrs ^or tiie svaiuation been secured? For example: 



Yes No Rank of importance 

□ D 2.1.1 Local schoo! boards? 

Uj □ 2.1.2 Local advisory committees? 

lJ Q 2.1.3 Communitv le«iers? 

2.1.4 State boards of education? 

2A,b Fedaral funding sponsors? — — ^ 

LJ O 2.1,6 State advisory counciis for career education? — 

□ n 2.1.7 Legislators? : 



n □ 2,1.8 
□ □ 2,1.9 



2.2 Proargm Staff and Pariieipsnts . Are program staff end participants involved in planning 
the evaluation? For example: 



n 




2,2,1 


Program staff {teachers, coordinators, administrators, etc.)? 


ri 

K 1 


n 


2.2.2 


Participants, usually students? 




n 


2.2.3 


Others involved in implementing the program (volunteers 








from busine», industry, labor, the professions, or govern- 








m«it)7 


! 1 




2.2.4 




d 


□ 


2.2.5 





2.3 General Pubitc> Will the public be kept informed of the goats and results of the evaluation? 
For example: 



( — 1 


n 


2.3. ! 


Parents? 


□ 


n 


2.3.2 


Other concerned citizens in the local community? 


n 


n 


2.3.3 


Professional colieagues in career education and evaluation? 


a 


i J 


2.3.4 




n 


□ 


2.3.5 
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1 The first group on the audience checklist in- 

2 J Decision Makers and Opinion eludes decision makers with positions pf for- 

Leaders, Have the sarK:tion and support maf authority as welJ as those persons stra- 

of the relevant decision makers and tegicaliy located at lower levels. Members of 

opinion leaders been secured? this group include local school boards, advisory 

1 committees, state boards of education, state 

advisory committees for career education, com- 
munity leaders, administrators, federal funding sponsors, and legislators at the local, state, or federal 
levels. This group usually repre^nts the highest priority audience^ especially if the focus of the evalu- 
ation IS on the sum worth of the program. 

A study of twenty federal health evaluations was conducted in order to identify f rctors that increase 
the use of evaluation results (Patton and others, 1973). The study revealed that the single most im- 
portant factor was the "personal factor/' The personal factor is made up of "leadership, interest, 
enthusiasm, determination, commit<,ient, aggressiveness, and caring about the evaluation (or program)/' 
In other words^ when decision makers and other relevant audiences were involved in thinking through 
and designing the study and were interested in and committed to the study, evaluations had an impact. 
' When these personal fetors were absent, there was a marked decrease in impact. Individuals who care 
^ about seeing evaluation results utilized should take seriously their responsibility for identifying relevant 
decision makers. This means finding a few decision makers and other strategically located persons who 
have a genuine interest in evaluation information and in the program, who know what questions they 
want answered, and who know how to use the evaluation information once findings are available. 
Evaluation should be conducted with these specific people in mind. This is the key to success. 



^ ^ 

2.2 Program Staff and Participants. 
Are program staff and participants in 
volved in planning the evaluation^ 



The second group of users of the evaluation 
results include program staff and participants. 

1 those who are accountable for implement- 
ing the program, such as administrators, 
teachers, counselors, other learning facili- 
tators, coordinators, etc. 



2. those who heip implement the program— such as resource persons from business, 
labor, industry, the professions, and government— but are not directly accountable 

3- participants in the program 



Program staff are frequently asked to use the results of the evaluation to make changes and improve- 
ments in the program. Program staff are also likely to be the most threatened by the evaluation. 
During the initial planning phase and throughout the evaluation, staff and participants should defi 
nitely be involved in; 

• reviewing the evaluation plan to insure its congruence to actu -l program activities 

• suggesting additional important outcomes that the evaluation should assess 
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• i'nteractins with the evsbator to voice concerns, constraints^ and limitations that should 
be acknowledged in the ^valuation 

1 

• reviewing evaluation insthinidnts for their fairness, comprehensiveness, and sensitivity in 
measuring Important asp^jcts of the program 

• reviewing drafts of evaluation r^orts 

These people turn tiie key. If tiiese individuals are involved in designing, critk|u!ng, and imple- 
menting the evaluation, they will bp more likely to accept, believe in, and use the results of the evalu- 
ation to improve programs 



2.3 General Public. Will file general 
public be kept informed ^e intents 
and remits of the evaluation? 



The third group of users of the evaluation in- 
cludes parents, concent citizens, professional 
cofl^i^es in career education and evaluation, 
and o^ers who are interested but not directly 
in\mh^ in die program. The evaluation re- 
y suits may serve several purposes with this group, 

including: 

• keeping the public informed about the goals and progre^ of career education 

• keeping colleagues informed about the progr^, strength, and weakness of your.program 

This group usually will not provide much input into the content and method of the evaluation; how- 
ever, their opinions about the program may influence their future support of and involvement in career 
education and thus indirectly influence the future of this particular program. The lock is open. 



Summary 



There are many potential audiences for an evaluation. The evaluation will benefit if you identify 

these audiences and make decisions about which of them are most important to involve in designing 
the evaluation. Knowing who the evaluation should serve will help you decide what kind of informa- 
tion to ask about the prc^ram. 
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3, Questions. What specific questions should the evaluation answer? 



There are many^potential questions that could be answered through an evafuation of a career 
education program. In fact, there are more questions than could possibly be address^ through any 
single eva||^ation. It therefore becomes essentiai to decide which evaluation questions about your 
particular program are most important and most practical to answer. The checklist below divides 
potential questions into seven major categories Examples of specific questions tor each category are 
listed in the mini-checklists that follow. 



QUESTION CHECKLIST 



Instructions: DecidsLwhich types of questions yo^^aluation will answer by assigning a percent 
to each of the following categoric of questions. The percents should total 100 per- 
cent and should reflect the relative importance of the categories. If you think that 
any of the question categories are not apprcHiriate, a^gn them 0 percent. 

» 

% 3.1 immodiate Student Outcomes. Questions about changes in students' knowledge, 

skills, attitudes, or behaviors resulting from a career-education expsri^tce (e.g.. 
Increased knowledge about careers, improved job-seeking skills, improveH attitude 
toward vmrk, etc.\ 

% 3.2 Long-Term Student Outcomes. Questions about die impact of career education on 

students' career development after they leave school (e.g., job placement, satisfaction, 
retention, advanc^ent, etc.) 

^% 3.3 intended Program Outcomes. Questions about change the program is designed to 

bring about ^at are broader than individual student '^tcom^ (e.g., irKireased com- 
munity involvement in the school, increased student involvement in community- 
based work experience, etc.} 

% 3.4 Side Effects. Questions alsout unintended or "spin-off" effects that the program 

may have on the school, students, teachers, parents, or the community (e.g., decreased 
local youth unemployment, decreased absenteeism, increased teachers' job satisfac- 
tion, etcj^ 

% 3.5 Prc^ram Characteristics. Questions about program resources, processes, and partici- 
pants (e.g., number and type of students involved, cost, instructional proce^, etc.) 

^% 3.6 Program Standards. Questions about how well the program meets standards (e.g., 

reasonable cost, relevance to needs, current content, absence of stereotyping by sex 
role, etc.) 

% 3.7 Audience Judgments. Questions about the judgments of various audiences on the 
program (e.g., students, teachers, parents, employers, etc.) 

% 3.8 Other. 



100 % Total 
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Examples of evaluation questions for each of these seven categories are provided in the foHow- 
ing seven mini-checklists. These questions are meant to stimulate your thinking about the many 
possible questions that could be answered tfirough an evaluation. Space is provided on each list for 
you to add additional questions. 



MINI-CHECKLISTS OF EVALUATION QUESTIONS 



Instructions: Rate the importance and fe^ibility of answering question from higher 

to lower where 5 « higher, 4 « moderately high, 3 « moderate, 2 * moderately 
low, 1 === lower. 



Enter your rating in the sp^es provided on eitfier side of 
lists. An example of the rating is provided below: 




Ions in the following check- 



EXAMPLE 



Importance 
4 



Do students in the program have improved 
knowledge of careers? 

Do graduates of the program have improve 
career satisf^tion? 

What is the cost of the program per pupil? 



Feasibility 
4 

2 



Interpretation: 

Those questions which are rated higher in importance (4 or 5) and wfwse feas'billty of being 
answered are at least moderate (3 or higher) are the most ^propriate to address in the evaluation of 
your program. 
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3J immediate Student Outcomes, 
Should th« a^aluatkm answer questions 
about changes In sUicients' knowiedgo, 
skiHs, attitudes, or bf^aviors reMJiting 
from a caraer^ucaticm experience? 



Do students in the program have improved/ 
incre^ed . . . 

Irhportance* Feasibility* 

....^^ career/job knowledge? -.p,,.^^ 

career decisions? ^ ..-.^.^^ 

decfsioh-making skills? 

job hunting skills? 

attitude toward learning? 

desire to work? 

self confidence/maturity ? 

interpersonal skills? 

self-understanding in relation to careers? 

specific occupational skills? 

competence In basic academic skills? 

work habits? 



awareness of opportunities for continuing 
their education? 

productivity in use of leisure time? 

awareness of means to change career 
options? 



*5 = higher, 4 = moderately high, 3 = moderate, 2 = moderately low, T = lower. 
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3.2 Long-Term Student Outcomes. 
Should the evaiuatiofi an^vw questions 
about th« impact of caretr education on 
students after they feave school? 



Jmport®tce* 



S 



Do graduates of the program have 
improved/increased . . . 



career identity? 
career placement? 
career satisfaction? 
job performance? 
Job retention? 

career mobiUty and ^vancement? 
salary levels? 

placment in career-related 
educational profirams? 

self-identity? 

self and social fulfillment? 

involvement in continuing or 

recurrent education? 

personal re^nsibility to 
employer? 



Feasibility 



*5 = higher, 4 = moderately high, 3 = moderate, 2 = moderately low, 1 = lower 
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3.3 tntendgd Program Outcomes. 
N ^oufd the evaluation answer questions 
\ about char^ the program Interxis to 
Ving about that are broader than in- 
dividuai student outcomes? 



Has the program increased/improved . . . 

importable* Feasibility* 

coitaborati\^ relationships with the em- 

ployment community? 

community awareness of career education? 

parental involvement in career «juc8tion7 

infusion of career-education conc^ts into — — . 

academic subjects? 

unpaid "career-exploration" work — 

experiences for students? 

instructional materials and supplies 

for career-education activities? 

f '^'jcational personnel's competenciM 

carrying out their roles in cai^r 
education? 

plans for coordinating and implementing 

comprehensive career education programs? 

career guidance, counseling, placement, 

and/or folio w-up services? 

/ 
/ 



\ 

\ 

^ ■ - \ 

5 - higher, 4 = moderateiy high, 3 - moderate, 2 = moderately low, 1 = \^ 
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3.* Side Effocts. ShouW the i^aluatfcjn 
enswer questions about unintendad 
or "spin off" effacts? 



Has the program . . . 

.Importance^ \^ ' FeastbilSty* 

(m:re£»ed/reduced absentraii^, tardiness, ■ 

or truancy? \ 

rai^/dociined achievement t«rt iicores? 



improved/lowered performance in rtpn- 
caroer oriented school subjects? 

increase/decreed vandalism, disruption, 
discipline probl^s, or aicohol/drug 
abuse? 

increased/decreased parental involvement 
in the school activities (PTA, etc.)? 

incrpased/decra^ed community »ipport of 
the Khool thrmigh bond issu^? 

increa^/decreased local youth ^ploy- 
ment? 



•5 = higher, 4 = moderately high, 3 - moderate, 2 » moderately low, 1 «» lower. 
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3.5 Pfogritm Chafctwittict. ^KHtW 
the ev«lu«tion wisvvtr qutstions ibcKtt 
program retourcts, groom(f$t tnd 
parttcipwits? 



Importance* 



What art the program's . . . 
purpoifii? 

costs (total and p&t pupiO? 

number and type of students s^v#d7 

type »id level of community invoiv^nent? 

cunrtculum mat^ials? 

instructional processes? 

staffing requirements? 

time requirements? 

in-service requirements? 

provistcms for populations with spectat 
needs Ca.g., handicipped, biiinguai, 
dis^vantaged, minoriti^, etc.)? 

provisions to promote sex fairness? 



Feasibility* 



•5 = higher, 4 = moderately high, 3 = moderate, 2 = moderately low. 1 lower. 
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3.6 Program Standgrds, Should the 
evaluation armm questions about how 
weif the protfrem meets standwdf for 
high quslity careef -education programs? 



Does the career-education program . . . 

Importance* FeasibHitv* 

have accurate and current con^t? 

focus on relevant topics? _____ 

have a reasonable ^>tt7 

avoid stereotypinf by sex role, race, 

age, or hamlicaps? 

have community a^^tance? ^ * «»...«_ 

have flexibility? 

use resources (time, people, dollars) 

efficiently? 

have comprehensiveness/variety/ 

range? 

he :jrovisions for continuity aftd 

faUowf-through? 

appear planned/coordinat«S/syste*natlc? 

^fwar balanced? 

appear forward-thinking/experimental/ 

creative? 

have evidence of sua:e$s/effecttvene»? 



*5 = higher. 4 » moderately high, 3 =» moderate, 2 » mcderaieiy low, 1 » lower 
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3.7 AydiefKse Judgmgntt. Should the 
evaluation coltoct th« |ud^ianti of 
various audi«nc8S mvofviKi in the 
progrnn? 



Importance* 



4 



Do students, teachers, perentt, employer! 
and ether relevant audlencei for the carett"- 
education program ... 



Feasibility* 



have a positive ettltude toward the 

program? 

express enthusiasm for t|ie prosram? 

perceive that the program has met 
their eKpectations? 

perciw /e the program as: 

useful? 

relevant? 

high quality? 

successful? 

timely? 

efficient? 



*5 » higher, A - moderately high, 3 » moderate, 2 » moderately low. 1 - tower 
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4. Pwcum. How will tii» avaiuottCHi be accompliittsd? 



To accomplish sn evaluation, you need information about: 

• Activities and tasks for conducting the evaluation 

• * Products the mfaluation wit! produce 

• Time lines for compieting tasks 

• D^tsion events in the evaluation prckess 

if the evaluation process is carefully planned in ativance, the evaluation has a better chance of pro- 
viding accurate and useful information. The prof checklist present four important fiictors to 
consider in planning your evaluation process. 

}/ PROCESS CHECKLIST 



Instructions: 
Yes No 

□ D 

n □ 
□ 



Check { / ) the apprc^sriate respfir^e to each question. 



4.1 Activities, Have the acti\\ties for accompiishing the evaluation been thought 
through and written dowr 

4.2 Products, Have the major pknlucu^ (e.g., instruments, reports, etcj to be 
produced through the.evafualion been identified and described? 

4.3 Ti me Lines. Have the various Waluation activities and products been carefully 
scheduled over the available time? 

4.4 Decision Events, Have review paints been built into the evaluation process for 
making critical decisions? 



4.1 Activities, Have the activities for 
accomplishing the evaluation b»n 
thought through and written down? 



we get this information? How might findings be 
the evaluation plan is implemented.. Instruments 



An evaluation has a beginning, a middle, and 
an end. In the beginning, the evaluation is 
pkinncd and given focus. Important questions 
—' Why are we doing this? Who should be in- 
volved? What do we really want to find out? 
How much money can vs^. spend? How can 
3?"— are asked af>d answered, fn the middle, 
developed. Information about the program is 
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coHected, analyzed, and interpreted. In the end, the evaluation re«jfts are communicated and used. 
The findings are summarized into technical and popularized reports and disseminated. The findings 
may then be used to make decisions, improve the program, or provide evidence for accountability- 
Below, some activities and tasks typically performed in an evaluate ^n are listed. 

EXAMPLE EVALUATION At TfVITIES 

1 . Plan and organize the evaluation. 

- Determine what information should be collected, who will use it, and how it will be used. 

- Budget and assign staff for evaluation, 

- Prepare for evaluation by developing a scope of work, a schedule, and role responsibilities. 

2. Implement the evaluation, 

- Develop or select instruments for collecting the information, 

- Collect the information. 

- Analyze and interpret the information. 

3. Communirate and usp the evaluation results, 

- Build acceptance of the evaluation results. 

- Prepare evaluation reports. 

- Disseminate evaluation results. 



- Use evaludtfon results. 

I 



4,2 Products, Have the major products 
to be produced through the evaluation 
oeen identified and described? 



Three types of products are generally produced 
in evaluation studies: 



1. 



An evaluation plan which outlines 

in some detail the components of 
the evaluation: purpose, objectives, audience, questions to be addressed, procedures for 
acc iptishing the objectives, activities and tasks, decision points, time lines, budget, staf 
fing, role responsibilities, instruments, data collection procedures, data analysis procedures, 
repof ting procedures, and dissemination prcK^eduf^es. / 

/" 

Instruments for gathering information, for example: multjiite-choice knowledge tests, 
opinionnaires and rating scales, interview questions, schedules for systematic observation, 
and performance test: ' 

/ 
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RepcKts which present the findings, including technical reports and executive summaries 
for sponsors and d^ision mak^; popularized reports and news releases for participani^ 
coffeagyes, and the general public; and "in-house" improvement-oriented reports for proj- 
ect staff. 



^•3 Time Lines. Have evaluation activi- 
ties and products been carefully scheduled 
over the available time? 



Time tines are among the most critical elements 
in the evaluation process. The following con- 
siderations are g^neraUv important ones for 
establishing the occasions when evaluation in- 
formation is collected: 



* Baseline information. Time should be allotted for collecting information before partici- 
pants are exposed to the program so that changes in knowledge, skills, or attitudes can be 

evaluated, 

* Sufficient time. Enough time between pre test and post test for some changes in participants 
to take place should be allowed. 

Timeliness. The results of the evaluation should be available in time to use in decision 
making. 

* School schedules- Data collection activities should be timed to fit school schedules. 



4.4 Decision Events. Have review points 
been built into the evaluation process for 
making critical decisions? 



A decision event or "milestone" in the life of 
a project is a pivotal ooint that requires an 
information-based review because several de- 
cision alternatives are possible. These decision 
alternatives are usually of three types: (1) to 
proceed without changes, (2} to make minor 
changes, or (3) -to make major changes. 

[n an evaluation effort, decision-event reviews provide an excellent opportunity to have externa! 
reviewers or evafuation auditors review evaluations developed by in-house staff. As a device to lend 
credibility and objectivity to an evaluation effort, the involvement of externa! reviewers is less expen- 
sive than contracting the entire evaluation process externally. 

The remainder of this unit covers thbe topics: { 1 ) identifying and selecting decision events for 
review, (2) preparing for the review, and (3) conducting the review. For more information about 
decision-event reviews, see Walker, 1973. 



Identifying and Selecting Decision Events 

Important points in the life of a program should be identified as likeiy decision events. Some 
likely decision events in an evaluation effort would be: 
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• df^" drafting an evaiuaticn pfan imt prior to imDlementins h 

• aftsT drafting an evaluation instrument but prior to coHecting the data 

• after drafting an evaluation report but prior to dis^minating it 

Each of these dcfcision events is an important future-oriented step in an evaluation effort that could 
benefit from a reappraisal. 



Preparing for the Review 

A decision-event should include: 

1. claims for the product (e.g., plan, instrument, report) being reviewed 

i 

2. information that supports these claims 

3. decision alternatives (e.g., no changes, minor change, major changes) 

4. opportunity for specific comments and recommendations of the r^iewers 

5. a review process le.g., mailing out the product to reviewers, bringing in consultants for 
the day, etc.), a time for the review, and persons to conduct the review 

A sample decision-event review form for a career-education evaluation instrument is pre^nted in 
Figure 5. 

Figure 5 



Sample Decision-Event Review Form for an Evaluation Instrument 



Claim 


E\^idence 


Decision* 


Comments/Recommendations 
of Reviewers 


Congruence. The items are 
togtcally derived from the 
program objectives and 
activities. 


See instrument. See program 
objectives. 


B 


The items on job hunting emphasize 
different skrHs tfian the origmaf objec- 
tives in this area. 


Comprehen^fvene^. The 
items cover all significant 
aspects of the program. 


See iiistrument, A test blue- 
print was used in developing 
the items. 


C 


Additional items should be developed 
to measure affective outcomes, e.g., 
attitudes toward learning, desire to 
work, self confidence, maturity, self- 
understanding, and interpersonal skills. 


ImBflrtf^gfifi The items 
measure significant as 
opposed to trivial learnings. 


See instrument. An advisory 
panel of teachers and students 
reviewed the Instrument, 


B 


The items on decision-making skills are 
trivial. 


Cfarity. instructions are 
cSear, items are straight- 
forward and unambiguous 


Se^ tristrument. The tnstru 
men^ was pilot tested with 
8 group of students. 


A 


Instructions are excellent. Items are 
clear and simply stated. 



A - Adequate, no change 
B Minor changes 
C - Major changes 
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A decision-event review is based upon the ctaims or standards for the product fc^ins reviewed. 
Three subsaquunt units of this handbook preset standards for reviewing the three d^ision events 
described above. Standards for reviewing evaluation plans are presented in Unit 7. Standards for 
reviewing evaiuation instruments are pre%nted in Unit 8. Standards for reviewing evaluation reports 
are presented in Unit 11. 

Conducting the Review 

Reviewers may work on-site or in different locations, independently or as a team. In every case, 
the genera! charge is for the group to review claims and supporting evidence provided by program per- 
sonnel and to look for discrepancies between those claims and the actual project. R^iev^i^ should 
complete the review form and prepare written reports (before they leave the location if on-site) that 
include ( 1 ) their judgment of whether claims v^e achieved, (2) their rationale for reaching their con- 
clusions, and (3) ^ecific recommendations for improving the project. 



Summary 



The chart pictured in Figure 6 combines the components descrit^ in this section into 
one chart. Evaluation activities, symbolized by rectangles I 1 , are scheduled over an 1 Vmonth 
time period. Evaluation products, symbolized by circles Q ' decision points, symbolized by 
triangles , are drawn in at the point in time when they are scheduled to be completed. 
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Figure 6 

Chart of Evaluation Activities, Products, and Dectsiom 



1 2 3 4 5 

1. Pi^n and Organize the Evaluation 



1.1 



1.2 



A® 
1.3 I 



8 



9 



— I — 
10 



11 



Monttis 



Z implmient the Ev^ation 




Communicate and Use the Evaluation B^its 



3.1 



- Evaluation Activities = 



t. Plan and Organize the Evaluation 



1.1 



1.2 
1.3 



Determine what information should 
be coilected, who will use it, and how 
it W! i be used. 

Budget and assign staff for evaluation. 
Organize for evaluation by developing a 
Kope of work, a timeframe, and role 
re^onsibilities. 



implement the Evaluation 

2.1 Develop/select instrumentCs) 
for collecting the information 

2.2 Collect the information. 

2.3 Analyze and interpret the information. 




3. Communicate and Use the Evaluation Results 

3. 1 Build' acc^tance of the evaluation results. 

3.2 Prepare evaluation report. 

3.3 Di^eminatti evaluation results. 

3.4 Use evaluation r^lts. 

Evaluation Products * ^ 

1. Evaluation plan 

2. Evaluation instrumentCs) 

3. Evaluation reportls) 

Decision Points « /\ 

1 . The evaluation plan is reviewed pnor to 
tmpl^/nenting the evaluation. 

2. The evaluation instrumentCs) is reviewed 
prior to collecting the da^. 

3. The draft of the evaluation r^ort(s} is 
reviewed prior to dissemination. 




B. Staff . Sliould some type of ex|i«t evaiuation m ici» be soufiht? 



The (i^Gpte who plan and impiemant an evaluation are essentia! to its success. The type of 
staff used in an evaluation Influences the cost, qusHty, credibiiity, and usefulness of the evaluation 
r{>sults. This unit pr«ientii smne alternative ways to staff for evaluation and some guidelines to local», 
select and u% evaluation services. When pcmible the evaluator (if otiter than pfo|ect steff } should 
be involved in the project as early as po^ble— perhaps even consenting to write tt% evaluation section 
of the pr(^^{. The staffing chscktfst pr^en^ four important questions about your evaluation 
staffing. 



STAFFING CHECKLIST 



Instructions; Check i <f ) the appropria^ response to each question. 



Yes 

n 
□ 

□ 

□ 



No 

□ 
□ 

□ 

□ 



^■^ Credibility. Will the Individuals who conduct the evaluation be perceived 
as credible? 

5.2 Cost-Effective Staffing. Is the staff ing for the evaluation cost-effective? 

5.3 Qualifications for Evatuatot^. Do you know what qualifications are needed 
in an evaluator for your program? 

5.4 Locating Staff . Do you know how to locate ccmipetent evaluation services? 



5. 1 Credibility. Will the individuals 
who conduct the evaluation be per- 
ceived as credible? 



Credibility of an evaluation is enhanced when 
the evaluator has 

• independence from the program, and 

• experti!»e in evaluation 

Evaluators should be protected from roles and situations that hamper their objectivity and ultimately 
their credibility. There are three common pitfalls that hamper the credibility of evaluators. These are: 

* Role conflict. For example, a per^n who has helped to develop a program is asked to 

evaluate it. 
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Conflict of fntiHiesL For example, a prof^onat colleague or personal friend is asked to 
conduct the evaiuatlon. 

Cooptation. For example, the evaluator's job security or status would be threatened by 
negative evaluation results. 



5.2 Cost-effective Staffing, fsthe 
staffing for the evaluation cost- 
effective? 



Cci«t and credibility are iwo key dimensions 
to consider when staffing for evaluation. It 
ts desirable to set up organizational arrange- 
r ients that provide maximum credibility at 
minimum cost 



A "ball park" estimate of the cost of evaluation is 10 percent of the total project budget. 
However, the cost of evaluations can differ widely, depending on the scope of work and type of 
staffing. 

More credible evaluations tend to be more costly because they typically involve longer time 
lines and the use of personnel with experti^ in evaluation. However, there are compromises that 
keep quality high arid costs low to mcKierate. The^ compromises generally combine the independ- 
erK^e and expertise of external evaluation with the cost-saving features of internal evaluation. Figure 
7 illustrates the relative cost and credibility of several ways to staff for evaluation, and the pros and 
cons of these methods are presented on page 55. 



Figure 7 

Credibility and Cost of Optional Ways 
to Staff for Evaluation 



CREDIBILITY 


High 


COST 
Moderate 


Low 


High 


An external evaluator is 
contracted for at least 10 
percent of program txrdget. 


Independent auditor verifies 
the design and implementation 
of the evaluation at key de- 
cision events. 


External reviews are 
systematically used at 
key decision events 
throughout the project. 


Moderdta 


Internal project staff with 
competence in evaluation 
are assigned major responsi- 
biiity. 


Evaluation consultant provides 
technical assistant in desiqn- 
ing an evaluation which is im- 
plemented by project staff. 


External reviews v e used 
at the end of the project 
only. 


Low 


Internal prxsject staff are 
assigned major responsi- 
bility to evaluate but tack 
expertise in evaluation. 


Internal proiect staff without 
evaluation expertise do the 
evaluation informaffy. 


No one is assigned rt sponsi- 
bility for evaluation. 
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Some pros ami cons of the» stsffing options are highlighted below. 

Pro Staffing Option Con 



+ cheap 



no assigned responsibiiity 



— nothing gets done 



+ are familiar with the 
project and the s^ttif^ 

^ know the pe<H3ie and 
the political constraints 

have low cost 



assign re^onsfbiiity to 
current project staff 



usually expertise in 
evaluation 

may experience role conflict 
or cooptation 

tack credibility 



+ have expertise in evalu- 
ation 

+ are famiriar with the 
project and the setting 

+ know the people and the 
politics 



hire evaluation expertis} as 
part of project staff 



4 have credibility 

have expertise 

+ are less susceptible 
to role onflict and 
cooptation 



use externally contracted 
evaluation services 



high in cost cost 

may experience role con- 
flict or cooptation 

have only moderate 
credibility 



moderate cost 

may lack familiarity with 
the project, the setting, the 
peopie, the (K)litical con- 
straints 

may be too 'Ivory tower" 



+ have checks and 
balances 

+ have credibility 

+ have expertise 

-t- have familiarity with 
project, etc. 



use a combination of 
external evaluation services 
and internal project staff 



may range in cost 
from low to high 



The optimal staffing pattern for evaluation uses some combination of external evaluation rM- 
vices and internal project staff. Several options are possible, for example: Have an independent third 
party conduct an audit of your evaluation (guidelines for using auditors are described on pages 28 
to 30); bring in an evaluation consultant to design an evaluation which is implemented by project 
staff; collect advice from external reviewers at key decision events \n the project ^guidelines for 
conducting decision event reviews are discussed on pages 49 to 51). 



~ I There are many important qualifications to 

5.3 Qualittcattons for Evaluators. consider v^en seeking evaluation staff or ser* 

Do you know what qualifications vices. You can use the staff ing-qua! if ications 

are needed in an evaluator for rating scale below to rate personnel that you 

your program? are considering for evaluation roles. Simply 

^ — I circle the most appropriate rating for each 

characteristic and complete a total score. 



High Medium Low 



STAFF QUALIFICATIONS RATING SCALE 

M@thodoiogical — 

1 . Knowfedge of research: review and synthesize literatare, com:eptualize reMarch 
problems, select inquiry strategies, formulate measurable questions to be ar^^vered, 
specify evidefKre necessary to answer the questions, apply research designs jirfiich 
control threats to validity and select samples, 

y 

2. Knowledge of innovations in evaluation: knov^ evaluation theory and models, 
use evaluation techniques from different disciplines, develop evaluation chtsigns, 
translate t^^oad goals into me^rable objectives, assess needs, develop standards 
or criteria for judging program wortn, r^Kirt evaluation results in a useful ^>d 
timely manner, develop recommendations^, and administer evaluations^ 

3. Knowledge of educational measurement: locate and select tests, ctovelop cognitive 
tests, attitudes scales, interview schedules, surveys, opinionnaires, and other meas- 
uring devices, and validate ir^truments. 

3 2 1 * 4. Knowledge of statistics and data processing: choose and employ appropriate 

statistical psts, use canned computer programs, develop computer programs, use 
computerfa d computer related equipment design and mana^ data banks, and 
interpretirie lesults of data analysis. 

3 2 1 6 Familiarity with the background, operations, substance, political factors, and 

constraints of the program. 

3 2 1 ** 6, Knowled^ of career education. 

Personal 

3 2 1 * 7. Freedom from vested interests in the outcome of the .rvatubtion. 

3 J 1 8, Fundamental agreement with the philoKJphy of the program to be evaluated. 

3 2 1 * 9. Reputation and status in the pfofessionalW»^<^«^^^«tV' 

3 2 1 **10. Trust and respect of the program staff. / ^ 

^especially important for sumrrrative evaluations 
**especislly important for formative evaluations 

Some rough guidelines for interpreting total scores on the staff qualifications checklist are presented 
below. However, remember th. c these qualifications may vary in importance for the evaluation of 
your particular program. 

3a27 26-23 22-19 18-15 Befow 14 

Excellent Very Good Good Fair Poor 
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j Csre^-'Ciducation project directors may.f ind it 

5"^ Locatirtg Staff . Do you know _ difffcuit to know when they are hiring a pro- 

how to locate competent evaluation fesslonaily competent mvi ethtc^sily responsible 

services? " evaluation consultant. Uppitt (tS74) states 

— ■ — J that "many a manager has soured on the use 

of consultants because of experience with in- 
adequate results, lack of client readiness, inappropriate ethics, hieh costs, and minimum contritMition 
to problem solving." There have probably bm\ even more sour experiences with extmtai evaluates 
than with other types of consultant. Yet, as explained in Section 5.t, outside consultants provide 
an independence and expertise titat are often essential for obtaining credible evaluative information. 
One solution to the dilemma li^ in learning how to select the right consultant Six recognized meth- 
ods of electing a con^ltant {Lippitt, 1974) are as follows: 

1 . The client should determine the nature and scope of the project prior to negotiating with 
prospective consultants. 

2. The client should review qualifications and experience of various consultants in relation to 
the project anc should make a short list of consultants qualified for consideration. 

3. The client should meet with each of these consultants to discuss the project and their ap- 
proach to it, and should ask each of them to submit a one or two page prospectus and 
budget. 

4. The client should check in depth the references of those consultants being seriously con- 
sidered. 

5. The client should then study the prospectuses in terms of each consultant's understanding 
of the problem, approach, probable benefits, costs, and particular ability to meet the re- 
quirements of the contract. In analyzing this information, the client should carefully 
consider the qualifications of the personnel who will actually be working on the project 

6. The client should base the final selection on a careful weighing of the at ove factors. Final 
negotiations should follow the selection of one consultant. 

Martha Williams at The Network developp ' a Program Evaluation Planning Packet (1976) for 
project managers which outlines ten tips for •„ jmers of evaluation consultant services. These are: 

1 . Solicit recommendations for evaluation consultants from other consumers-project direc- 
tors, schcKjl district ^ministrators, state department of education personnel, or a human 
resource file. 

2. Find an educational evaluator, not simply a person with strong statistical or computer 
background. The consultant should have experience in evaluating educational projects 
(both process and prtxluct evaluation) and be familiar with design constraints and alter- 
natives. 

3. When you have identified candidates, ask them for names of other clients they have served. 
It's wise to talk with a few others who have worked with the consultant t^ learn about his 
or her jtyle, expertise, and ability to work within a schedule. 
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4. Choose a consultant easily accassibie to the project A consultant who is geographlcalfy 
dose to the project site can be available tor meetings, on-site data collection and ri^orting, 
and other aspects of the f omiative eviiluationi process. Al^, keepir^ travel costs down 
helps to make the evaluation OHt-effective. 

5. Determine the cost of evaluation in advance, based on what is budgeted or what can be 
transferred to an evaluation line item. A good rule of thumb established the cost of ^af- 
uati(»i betweeii three and eight percet^t of the total project budget. 

6. Negotiate wltii the coniuiltdnt tor frequent on-site visits to discuss procedures, interim 
results, and problems which arise. ^ 

7. Contract carefully with the consultant. Wh^ formulating a contract with an outside 
evaluation consultant, the following thinp should be considered: (a| who has title to 
the data— make certain that the project, not the evatuator, has tiiat title; {h) the exact 
terms of the evaluation-v^at is exr^ted of whom, when^ (c^ the number of days on- 
site; (d) the itemized budpt for the evaluation; and (el a cam^Hation clause allowing 
S&day notice for cancellation of the contract for both the project arrd the evaluator. 
The contract should be in writing, signed by both client and consultant. 

3. Be sure the consultant is introduced to and accepted by key individuals involved in the 
evaluation. If the consultant meets resistance in obtaining data, the evaluation will be 
weakened. 

9. Consider having the consultant conduct in-service sessions for project staff and teachers 
to acquaint them with the purposes and procedures of evaluation. Be sure staff under- 
stand the way in which the results of the evaluation will be used. 

10. Establish the evaluation design as a cooperative effort involving the project director, the 
project staff, and the evaluator. This involvement will increase the useftrinfss of the study 
and results, take into account reality factors and limitations, and insure that the consul- ' 
tant's interests and orientation don't overly influence the design. 
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6. RttiKNriijMWty/Auttiofity^ Kavt ^ program stiff ar>«l 
•vahMktor Igivtd iikptetiva tasks? 



Th0 reSBtioi^'^hip b«tw0«n an ovsluator and the program itsff (S a critical afammt in the 
of m evaluation iVioit of ut are sorrMw^t threatenid by ^ prospect Pf ba^ng ttvalua^. We may 
become especiaUy wary when th« evaiuator is ext^ttal to th« project Thms ti a fear that th« evaiu* 
8tor may not fully undfifstand the pro|ect-ils history, setting, constraints, philosophy, activities, 
and outcomes. There is also a fear the evaiuator will impose his/her own values tm the program or 
wiii have ■ different phiioiophy of career eckication than that emphasized by the progrim. Another 
common ^ar ti that the evaiuator will iHimie the i^rfact mearch design and thus min the mte im> 
pact of the progriun. 

The evaiuator also ha& fears. The evaiuator may fear that his/her hands will be tied by the proj- 
ect staff and thus be prev^^ from taking an objective view of the program. TYte «valuator may to 
that l«s than pmitive resjlts akxmt the program may cause problems or that prdsirig items may be 
deleted from Instruments. 

Some of the fears can be allayed by clarifying roles, responsibilities, and authority for the pro- 
gram staff and the evaiuator at outset of an ^iuation effort. The RmfHsmibility/Authority 
Oh^cklist mimmarizes KHne areas when clarity artd agreement are esamtial. 



yl RESPONSIBILITY/AUTHORITY CHECKLIST 



Checic ( ) the appropriate responi^ to each question. 



6.1 Evaluator-Administratof Responsibilities. Have responsibilities been clearly 
defined for the administrators and evaluators? 

6.2 Evatuator-Adr..inistrator Authority . Is there agreement about the evaluation 
decisions and who will make or be Involved in them? 

6.3 Communication/Protocol. Have the channels of communication among the 
project sponsors, project director, project staff, and the evaiuator been es- 
tabllsl^? 

6.4 Dis^reemen^, Are there procedures for reiving dia^reements between 
program steff and evaiuator? if and when they arise? 
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instructions: 

Yes No 

□ □ 

□ D 

□ I 

□ □ 




6.1 gvfltyw-A^n^^n'mtgr R^nii- 

biti^tes. Have respon$lbiiittes been 
dearly defined for the administration 
and evaluations? 



''An evaluator is a resource and should not be 
supi^ised bv the administrator. The relation- 
ship should be much like that of a doctor* 
patient or lawver ^iient--i%hiie the client main- 
tains the final veto power, the evaluator has 
discretion and freedom to propose and even- 
tually implement methodological practices. 
Obviously, these must be agreed to by the client before they are initiated/' CAdams and others, 
1976) 

Decisions about who ^ould take responsibility for various evaluation activities should be made 
at the outset of the evaluation process. These decisions should be based on the ansvwrs to such ques- 
tions as: 

• Would the activity benefit from evaluation expertise? 

• Should the activity be performed by an independent a^ti;tY to enhance the credibility of 
evaluation results? 

• Should the activity be performed by someone who is intimately familiar with the project 
to insure that the evaluation will provide useful information to relevant audiences? 

• Should the activity be performed by project staff to conserve re.vources? 



A sample division of the responsibilities between evaluation and administration has been drawn 
up using the above guidelines. Figure 8 on page 61 lists some activities typically performed in an 
evaluation and a potential division of responsibility between evatuators and administrators for each 
activity. 

There is an important balance to maintain between the respective roies of evafuators md program 
staff. Program staff should help insure that: 

• the program is evaluated fairfy 

• the evafuator's values do not unduiy influence the direction of the evaluation 

• checks and ^aiances are used in the evaluation process to incorporate alternative points of 
view and prevent a one-sided evaluation 

• there are frequent opportunities for interaction between teachers (or other program staff) 
and the evaluator in order to report on progress and share concerns 

On the other hand, the evaluator must help ensi e that: 

• the evaluation of the program is as objective, impartial, and unbiased as possible 

f 

• evaluation instruments aliow the respondent to provide honest rather than "socially 
acceptable" responses 
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• the evaluation fnitruments ask probing qu^ksns 

• the evaluation instruments delve into the wsaktmsas as w^l a$ the stiwng0$ of the program 

• negative findings are rapor^ honestly 

• findings are carefully qualified and are free of exag^ration or overstatement 

Fi^re 8 

Evaluator-Administra^ RoponiilMlEtias 
Evaluation Activity ^ Rtt»tiontibititv 

1. Plan the Evaluation, including 

- Determing what information should be collected, who will EA^ 
use it, and how it will im used 

- Budgeting and assigning staff for evaluation A 

- Organizing for evaluation by scheduling ^tivities and products AE 
and determining role responsibilities " 

2. Implement the Evaluation, including 

- Developing or selecting instruments for gathering information EA 

- Collecting the information Ea 

- AnuSyzing and interpreting the information Ea 

3. Communicate^ and Use the Results 

- Building acceptance for the results Ae 

- Preparing evaluation reports Ea 

- Disseminating evaluation result Ae 

- Using evaluation results Ae 

•Key 

A ^ Administrator 

E Evaiuator 

— - Placement firit indicates major responsibility; placement second indicates less responsibility, 

- = Capital letters Indicate major input: small letters indicate less input. 
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6.2 Evatuator— Administrator Authority . 
h there agreement about the evaluation . 
decisions and who wiit make or be in- 
volved in theni? 



Clarifying decisions and roles can be facilitated 
by constructing a decision-by-person matrix 
V that specifies what person: 



program evaiuator / 
program director 
project monitor 

advisory committee (chairperson) 
school board (chairperson) 



signs off in what way— 

• approves/disapproves 

• reviews, or 

• is informed 

on what decisions— 

• objectives for the evaluation 

• evaluation instruments 

• final version of the evaluation report, or 
m distribution of evaluation reports 

Some of the critical decision areas which should be included in a decision-by person matrix are 
highlighted below. 

• Editing instruments. Who decides what rterqs are included or excluded from evaluation 
instruments? 

© Editing reports. Who decides what *formation is included or excluded in the evaluation 
reports? 

• Disseminating reports. Who decides what audiences may receive the evaluation reports^ 

• Passing judgments. To what extent will the evaiuator interpret the results and draw con- 
elusions about the sum worth of the program? 

A partial example of a decision-by-person matrix is presented in Figure 9, 



Figure 9 



Partial Example of Decision-by-Person Matrix 



Person* 



Decision 



Approves/ 
Disaooroves 



Reviews 



Is informei 
Aboiit 



Final editing of 
evaluation instrument 



PD 

PE 



AC 
SB 



Reiea^ of 
evaluation report 




AC 



^Person: PD-Program Director AC^Chairperson of the Program Advisory Committee, SB ^ Chair 
person of the School Board, PE-Program Evaiuator, PM=Project Monitor. 



6.3 C9^^^Mnig§tiQn/Pr(?tW<?t- Have 
channels of communication been es- 
tabiished? 



Should the evaluator have direct contact with 
the project sponsor or go through the project 
director? Should the evaluator have direct 
contact with project staff or go through the 
project director? yvhat protocol shcHild be 
followed in contacting various schools, class- 
rooms, etc. where data will be collected? It is essential that clear channels of communication are 
establisheil and that appropriate protocol is followed in an evaluation. 



6;^ Disagreements . Are there procedures 
for resolving disjigreements t^tween pro- 
gram staff ann ^valuators? 



Because evaluations are sensitive and value- 
laden, disagreements usually arise. Teachers 
may feel that their views have not been fairly 
represented in the evaluation. The project 
director may feel that the evaluation has 
been unfairly'^harsh on the prc^ram. The evaluator may feel that a group of items, althou^ contro- 
versial, will providfe valuable insights about a program. 

To avoid as many disagreements as possible, program staff and evaiuators ^ould maintain fre- 
quent communication. Two-way exchanges of Information afcKsut the progress of the evaluation de- 
sign, instruments, activities, programs, and new developments are essential. However, even with 
frequent communication some disagreements are likely to occur; perhaps the most common are 
riffs b^'een teachers and evaiuators. Because of the nature of valuation, some disagreements will 
not be possible to resolve to everyone's satisfaction. Neverthele^, strategies for arbitrating disagree- 
ments, resolving conflicts, and gaining consensus should be devised for use in these situations. 




7. Uniquam^. What unique features of caraer 
Miucation inftuence Its evaluation? 



Evaiuation has probieais and process^ that are common to all types of programs. However, 
career education has some unique features that must be considered in designing an evaluation of such 
programs. The Uniqueness Checklist ^mmarizes some of the unique features of career educatiqQ, 



UNIQUENESS CHECKLIST 



instructions: Decide whether the following features of career education have been considered in 
designing your evaluation by checking { y ) the appropriate box. 



Yes 


No 




n 


n 


7.1 


D 


□ 


7.2 


D 


□ 


7.3 


□ 


a 


7.4 


L] 


[□ 


7.5 



Career education is an evolving and elusive 
concept. Just when you feel you understand 
career education, the philosophies and trends 
shift. Consequently, many different frame- 
works, lists of goals and objectives, elements, 
and instruments for career education have been developed. There are also divergent points of view 
about what is and what is not career education as watl as how it should be delivered. The concept 
of career education varies from helping prepare learners for job roles to helping prepare them for 
roles in the family and in society, as well as for activities in their leisure time. The delivery of career 
education varies from infusion of career ^ucation concepts into the entire school curriculum to spe- 
cific career education courses and units. Career education may have as many definitions as there are 
teachers, counselors, and others involved In its delivery. Some career education programs have no 
underlying definitions, theory or philosophy to give the program direction. 

Because career education is constantly changing and often highly individualized, it is difficult 
to measure and summarize its impact on students. But the problems are not insurmountable. It is 



7.1 Evolving Definition of Career 
Education. 
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important to clarify ami delimit what you are trying to achieve through career education ijefore em- 
barking on an evaluation. Then a variety of evaluation approaches that measure the various facets of 
a career-education program can be quite effective. The areas of impact common to all techniques can 
then be summarized. Revievi^ again the variety of approaches described in Unit 1 ("Purpose") to find 
an approach or combination of ^prosches that will fit your unique situation. 



Most people from educators to assembly line 
workers to corporate presidents have an opinion 
about career education. And there are many 
different opinions about the rote of career 
education in our society. Some believe that career education is a process of providing systematic 
information and experiences to asist a person's career development. Others believe ^at career edu- 
cation is propaganda for capitalism and the protestant ethic of "hard work is its own reward." Still 
others believe that career education is creating unrealistic expectations about the pleasures and satis- 
factions to be derived from working when, in fact, much work is not pleasant. An individual's philos- 
ophy of career ed.ucation will strongly influence the way an evaluation of a career education program 
is conceptualized and received. Therefore, clarity and agreement about the underlying philosophies 
of career education among program staff, sponsors and evaluators is essential for an effective evalua- 
tion. 



7.2 Disparate Philosophies of 
Career Education 



I "Big criteria have little criteria that follow 
7.3 Illusive Criteria. ; aro ind behind mem." Career education opens 

j up a vast array of outcomes that schools have 

not been held directly «:countdble for in the 
past, in basic skills it has been enough to prove that schools have helped students accomplish minimal 
competency in the 3 R's. But in career education, proving only that kids have better self -awareness or 
career awareness may not be sufficient. Critics ask "so what?" If students have self -awareness, will 
they make more informed career decisions? If students make more informed career decisions, will 
they have greater personal and occupational satisfaction? There is an almost endless chain of out- 
comes, and it is necessary to identify the most important ones in order to evaluate vhether career 
education has truly made a difference on a student's career development. 



7.4 



Differing Theories of How Career 
Development Takes Place. 



Career Development is a complex and highly 
individualized process. We do not yet know 
enough about how career development oc- 
curs to measure it confidently. Although 
some authors claim to have developed tests 
or kits of instruments that validly meaajre career development, these tests are limited by our iim-. 
ited knowledge of the career development process. At this time, there are few valid and reliable 
measures of selected a^acts of career development. Therefore, claims about what a career educa- 
tion instrument is measuring should be carefully qualified. 




Career education often emphasizes affectiVie 
outccKnes, such as incr^s^ s^lf-swareness, 
improved work attitudes, clarified work values, 
and more productive work habits. Affective 
me^rement presents some unique problems. 
Affective outcomes are typically more diffi- 
cult to measure than cognitive or psychomotor outcomes. Less Is known about affective variables 
and how they are formed and changed. It is often difficult to parcel out the affective outcomes that 
the career education program has helped produce. Attitudes, vaiues, and appreciations are intimately 
linked to an indivkjual's personality and background. 

A person's total environment influences attitude, growth, and change. Thus, it is very difficult 
to determine when and how much affect "ve qualities are being influenced by career education. Chan^ 
in attitudes, values, and appreciations are difficult to detlct through paper and pencil evaluation 
instruments. 



7.5 Difficulty o f i ^sunnt| Affective 
Career £du{:at!3n Outcomes. 
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^ Wanntna Standreto. How do you recogniai a good tvttuction plan? 1 



This unit summarizes key ingredients in a good evaluation plan. It &ito outlines sortte criteria 
for assessing the ovt^li quality of an evaluation plan. Tmf tumnutry dtedclists are provided. The 
Checklist of Parts for an Evstuation Pian can be with the Chttddist of Cn'Wm to ass^ ^e over- 
all quality of your evaluation plan. It would be ^>propriBte to use either or both in a decision-event 
review {see Unit 4 pages 49 to 51 for more information). 

^f CHECKLIST OF PARTS FOR AN EVALUATION PLAN 

instructions: Rate your evaluation plan by clicking ( ^ ) dw appropriate response about the parts 
and the phrasif^ of your plan. 

mi Betttf hk>t Not 
Su^ Sta»tn«m Sta^ Applied 

Conceptual Plan 



8.1 


Statement of purpose? 


□ 


□ 


p 


p 


8.2 


Questions to be address? 


□ 


a 


p 


p 


8.3 


Audiences to be served? 


□ 


□ 


p 


p 


3.4 


Potential uses of the remits? 


□ 


□ 


p 


p 


8.5 


Overview of activities? 


□ 


p 


p 


p 


8.6 


Evaluation products to be developed? 


□ 


p . 


p 


p 


Managwnent Plan 










8.7 


Organizational locus of evaluation (e.g.. 


□ 


□ 


p 


p 




internal or external, line or staff)? 










8.8 


Policies and procedures affecting the evaluation? 


□ 


□ 


p 


p 


8.9 


Staffing? 


□ 




p 




ato 


Rote responsibility and authority? 


□ 


□ 


p 


p 


8.11 


Budget? 


Q 


□ 


p 


p 


ai2 


Facilities and equipment? 


□ 


p 


p 


p 


8.13 


Time lines for activities and procedures? 


D 


p 


p 


p 


a 14 


Decision events and standards for 


n 


p 


p 


p 




monitoring the evaluation process? 










Technical Design 










ai5 


Data gathering instruments? 


□ 


□ 


p 


p 


ai6 


Data gathering procedures? 


□ 


p 


p 


p 


ai7 


Sampling plan? 


□ 


p 


p 


p 


ais 


Data storage and retrieval procedures? 


□ 


p 


p 


p 


ai9 


Data analysis procedure? 


p 


p 


p 


p 


8.20 


Data interpretation procedures? 


a 


p 




p 


8.21 


Procedures for reporting die remits? 


D 


p 


p 


p 


a22 


Procedures for disseminating the results? 


□ 


p 




p 
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CHECKLIST OF CRITERIA FOR REVIEWING EVALUATION PLAf^S^ 



Instructions: Rate your evaluation plan against the following criteria by checking ( ✓ ) the appropriate 
response. 

Yes No 

lII □ 8.1. Scope, Does the range of information to be provided include alt significant as- 
pects of the program being evaluated? 

□ □ 8.2. Relevance, Does the information to be provided serve the information needs of 

the intended audiences? 

D □ 8.3. Flexibility , Does the evaluation plan allow for new information need: o be 

met as they arise? 

□ □ 8.4, Feasibility , Are the resources of time and money adequate to carry out the 

evaluation as planned? 

l1 n 8.5. Reliability . Wiu the information be collected in such a way that If someone 

repeated the study they would obtain similar findings? 

□ □ 8.6. Obiectivitv ^ Ha^e provisions been made to help eliminate bias ir data collection 

and processing? 

u D 8.7. Representativeness. Will the information collected accurately and fairly portray 

the program? 

G □ 8.8. Timeliness. Will the information be provided in time to be of uve to the audiences 

for the evaluation? 

LI LI 8.9. Pervasiveness. Will the information be provided to at! who need it? 

n n 8Jg. Ethical considerations. Will the evaluation guarantee confidentiality and pro- 
tection for those who provide Information? 

rj [3 8. . Protocol, Are conventional protocol steps planned for contracting people in 

the appropriate sequeni e and for following existino ooiicies and procedures? 

□ n 8.12. Security. Have provisions b^n made to maintain the security of the m^alua* 

tion data? 

O □ 8 J 3: Credibility . Does the design of the evaluation encourage trust in the results by 

relevant audiences? 

n n 8.14, Cost-effectiveness. Compared to its potential payoff, will the evaluation be 

carried out at a reasonable cost? 

•Adapted from James R. Sanders and Dean H. Nafziger, 0976). 
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IMPLEMENTING THE EVALUATION 



This part ot the handbook is brief becmjse there are numerous excellent resourcet available on 
imptennenting an evaluation. Implsffnenting an evaluation imiludes activitin such as: 

• developing, pilot testing, and rev iing instruments 

• selecting a research design 

• developing a sampling plan and data collection schedule 

• selecting appropriate statistics for analyzing the data 

• collecting the data 

• coding, keypunching or tallying, and storing the data 

• analyzing the data by hand, calculator, or computer 

• summarizing the data into tabi^, charts, etc. 



Basic introductory texts on r^earr.h, measurement, and statistics are the most comprehensive 
sources for learning more about thea \ >pics. Some recommended resources are: 

Research 

Best, J. W. Research in Edt cation (3rd edition) Prentice-Hall, 1977. 
Measurement 

Ebel, Robert L. Measuring Educational Achievement. Englewood Cliffs, New Jersey: 
Prentice-Hall, 1965. 

Thor ndike, Robert L. (ed.) Educational Measurement. Washington, D.C.: American 
Council on Education, 1971. 

Statistics 

Guilford, J. J. Fundumentaf Statistics in Psychology and Education. New York: McGraw 
Hill, 1965. 



This part of the handbook focuses on one of the most critical aspects of implementing an 
evaluation— insuri ng that the data collection instruments will provide accurate, useful, and com- 
prehensive information for making decisions about your pi-ogram. For more detailed information 
about developing evaluation instrunnents for career education, see 4 Cuida for Improving Locally 
Developed Career Education Measures (McCaslin and Walker, 1977), in this Careei' Education Meas- 
urement series, 



71 



9. InstnjfKient Standards. How do you recogniice a "good" evafuation instrument? 



This unit contains guidelines for deciding if jin instrutnent will ... 

• provide the information you and others most need to know about the program and 

• provide reasonably accurate information. 

Evaluation instruments can be very brc^dly divided into two categories: 

r 

1 . those that seek objective information, such as facts statistics, and direct measurement 
of knowledge, skills, or behaviors. 

2. those that seek subjective information, such as perc^tions of needs, opinions, attitudes, 
self-assessments of Interest or abilities, and ratings or program quality 

The following checklist is designed to help you review both types of instruments systematically 
in order to assess their quality. The checklist could also be used in a decision-event review by ex- 
ternal reviewers (see Unit 4, page 49 for more information.) 
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INSTRUMENT CHECKLIST 



instructions: Rate your evaluation instrument against the following criteria to determine its 
strengths and weaknesses. Check ( ✓ ) the appropriate box for each item. 

Yes No 

O □ 9.1 Relevance, is the instrument appropriate for the program? Do the items describe 

outcomes that are actually being sought in the program? 

n □ 9.2 tmportancfi . Do the items describe outcomes that are significant for students 

to master? 

□ □ 9.3 Practicality. Do the items describe outcomes the program can realistically change 

in the amount of time available? ( For example, some outcomes-self -corniept, 
work values, etc.— change more slowly than others. Some outcomes— career 
placement, satisfaction, advancement, etc.-may be tc 3 distant to be affected 
by the program.) 

n n 9.4 Measurabilitv. Is it feasible to measure the dimensions that the instrument is 

attempting to measure with the resources available? {Some dimensions, e^^eciaily 
in the affective domain, are more difficult and time-consuming to measure pre- 
cisely.) 
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Yes Mo . 

□ Q *9.5 B3lgrK;e/Cornprehenstvene$s. Do the items on the instrument represent a bal- 

ance of information about ail the important a^ects of the program? 

□ □ 9.6 Objectivity. Are the instructions and items worded tn such a way that people 

will respond honestly rather than in a "socially acceptable" way? 

□ □ 9.7 Attr^tivrene^. Is the instrument designed to encourage complete and thought- 

ful responses? Does it have complete and clear instructions, legible print, at- 
tractive layout, and reasonable length? 

□ Q 9,8 Ctarity , Can the items be easily understood? Are they free from ambiguity? 

Gl CD 9.9 Readability. Do the items usq as few words as possible to communicate? is 

the reading level appropriate for the individuals to be tested? 

□ □ * *9. 1 0 Challenge. Is the right answer so oiJvious that most students can answer it 

from common sense? Is the right ansvrar so obscure or tricky that everyone * 
misses it? 

n Q **9.1 1 Differentiation, If the test is norm referenced, has each item been tested 

to ensure that it differentiates between students with the highest and 
lowest ability? 

n n 9 J 2 Sex/Race Fai ness. Are the items free from stereotypes by sex or race? 

□ n 9.13 Manageability. Do your staff have the time and expertise to summarize and 

analyze the information being collected? <For example, responses to open- 
ended questions will take more time and experience to nterpret.) 



f I 



9.14 Validity. Has the validity of the instrument been systematically tested and 
report^ 



□ □ 9.15 Reliability . Has the reliabllitv of the instrument been systematically tested 

and reported? 

n □ 9.16 itern Analysis^ Has an item analysis been conducted and used to improve the 

test? 

LJ Q 9.17 Pilot Testing. Has the test beef tried out with a group of typical respondents 

and revised to include their feedback? 

□ LD 9J8 Usefulness. Has a use been determined for each piece of information to be 

collected. Have the items with no apparent use been eliminated or modified? 



*This criteria applies only to subjective items. 
** This criteria applies only to objective items. 



74 



ERIC 



COMMUNICATING AND USING EVALUATION RESULTS 



Evaluation is an arduous undertaking, $o it is unfortunate when evaluation findings are not 
used. As discussed in . the introductory section of the handix>ok, ibBW are various reasoos why 
evaluation results are not used.. The results may arrive too late for making decisions. The results 
may be too vague, complex, unclear, or redundant. The results may not be disseminated effectively 
through various channels, or they may not be ^ared in a manner that communicate what they mean 
and how to use them. 

This section provides guidelines and techniques for increasing the use of evaluation results to 
improve programs, make data-based decisions, and keep the public informed ot program processes 
and outcomes. 

This section contains four units: 

• Unit 10 provides ideas for dealing with the political factor? that affect use of evaluation 
results. 

• Unit 1 1 provides two checklists. One checklist is for reviewing the parts of an evaluation 
report for completeness of information. The second checklist provides criteria for 
assessing the {accuracy, utility, and feasibility of evaluation reports. 

• Unit 12 provides guidelines for sharing the results of evaluations with different audiences 
through popularized reports, multiple short reports targeted for different audiences, 
executive summaries, news releases, presentations, and other media. 

• Unit 13 provides advice on using evaluation rt^uits to bring about change. Strategies 

for overcoming personal and organizational barriers and ways to facilitate use of evaluation 
results are discussed. 



75 



I II • I I 

10. Politics, What political f«ton might affoct use of titt evaluaticm rMuits? 



Evaluations are in part political activiti^ and thus are susceptabte to political influents. 
Evaluations are conducted to help decision makers decide who ^ts what, so the political Influ- 
ences on evaluation whether »ibtle or blatant, are numerous. Have you cor^sidered the political 
factors that influence vour program and itii evaluation? Many may be unique to your situation; 
some, such as those presented in the following checklist, are widespread. Brickelt (197g{ has 
developed a series. of scenarios that describe political infiuertces on evaluation, and the following 
checklist is based on his work. 



POLITICS CHECKLIST 



Instructions: Consider each of the following factors an^ indicate whether it influences your pro- 
gram by checking { / ) the apprcHpriate box* 

Yes No 

□ □ 10.1 Potittcal motivation. Was the evaluation initiated for basically politically 

reasons? 

□ Q 10,2 Payoff . Does the payoff for coming up with positive findings overshadow 

the payoff for conducting an effective evaluation? 

i O 10.3 Megative findings . Have you thought about how you would handle un- 
favorable or nonsignificant findings about your program? 

□ n 10.4 Realisticcriterta, Is your evaluation using realistic criteria for success? 

□ Q 10,5 P^ision influences, ill other sources of ^formation besides the evalua- 

tion results be used to decide the future of your program? 

D n 10.6 Slanted evatuation. Is the evaluation being unjustly slanted in either a 

negative or positive direction? 
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10.1 i otlt|cat Motivation . 



Evaluations are initiated for a variety of 
political reasons: to app«B8 adversaries of 
a program, to reap a reward such as special 
funding or exemplary status, or to comply 
with requirements (even when there is no real interest in using the resultsK Evaluations are seldom 
initiated from a truly objective, apolitical point of view. 




The payoff for evaluators, program directors, 
and funding sponsors is usually to come up 
with /^/f/M9 evaluation results. Professional 
court^*K and a desire for continued business 
often make an evaluator just as anxious for positive results as tKe program staff. Evaluators have 
been known to sit on the sidelines throughout the evaluation voi(»l«slv cheering "Make a significant 
difference." Program directors and staff naturally want to see their program praised and evaluated 
favorably. Even project monitors and funding sponsors want positive evaluations. After fighting to 
secure funds for a program, helping to nurse it through the rouqh spots, and praising the program to 
constitutents, project sponsors are much happier when an evaluation reveals positive outcomes. 

Political pressures may encourage an evaluator to eliminate or play down negative findings in 
reports and thus present an inaccurate or misleading picture of the program. It is your role to facili- 
tate an effective and objective evaluation of your program. 



10.3 Negative Findings. 



A wise saying, first espoused by Rossi (1972), 
is "no good evaluation goes unpunished/' 
Evaluations can'usuatty be discredited on some 
basis: inaccurate data, faulty instruments, 
misunderstanding the program, incompetent interviewers, biased data interpretations, slanted writing 
tone, etc. If someone is not satisfied with the results of an evaluation, it is often easy to find weak- 
nesses in the report. 

^ Encourage your evaluate to provide specific and balanced information about the'^program's 
^ strengths and weaknesses. Ask for constructive feedback so the weaknesses of the program can be 
improved. 



10.4 Realistic Criteria. 



evaluated using student learning as the criterion 

training program to make significant, measurable differenced on student learning 
using this criterion would probably be disappointing. 



Evaluations are sometimes conducted with 
unrealistic criteria. For example, a program 
to provide in-service training for teachers in 
career-education infusion tactics may be 
It would be (difficult for even the best teacher- 

An evaluation " 



10.5 



Daci^ion Influences. 



Evaluation results are usually treated as only 

one basis for making decisions. Consequentty, 
a program with highly positive results may be 
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terminated becauije Its siubstaoce is not a high priority item in a given year. 'Another program may 
have h&d no measurable impact but receive double fynding after an especially effective testimony 
before legislators; 



10.6 Slanted Evatuation. 



findings. Given the state of the art, it i$ possible 
Two evaluations of the same program c^uld have 
negative. 



The art of evaluating career education is at 
an eariy stage of dwelopment. Bi^use the 
evaluation tools are Mmited, it h impossible 
, to have ciear^iut, totally decisive evaluation 

to use evaiiuation in some instances as a pawn. . 

directly opposing results— one positives, She other 
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11. Report Standards. How do you re^ogiiizs a g«^ w^ svaluatioii report? 



A ''good''evaluationreport can take a var iety of forms. 8t may be short or long, it may be 
communicated in writUng or through face*to*face dialogue. It r ay be highly statistical or primarily 
yescriptive and anecdotal in orientation. The type of report which is prepared depends on the type 
&of program, purpose of the evaluation, arvd type of audience to which the results are l^ing commu- 
nicated. 

"^is unit highlights the criticai components of an evaluation report, elaborates on the content y 
making up these components, and summarizes K>me criteria for juaging the overall quality of the 
report. 

No two evaluation repo. .a can be expected to have exactly the sanje format ard writing style, 
• but most reports should contain the following five romponents: 



Cover page 

Recommendations 

Introduction 



Evaluation procedures 
Results 



further information about each of these component* is'^provided below, (adapted from Adams, 1976) 



r 



--11.1 Cover Pane. 



The c^ver page may give a descriptive title to 
the evaluation study, such as "Formative 
Evaluation Report for the Career-Education 
Infusion Strategy Workshop/' If the report 
is informal rather than technical, then £ n informal title, such as "Results end Recommendations for 
the Career Education Infuslon-Strateg/ Aforkshop" may more appropriate. If for a lay audience, 
then a more "catchy" title may be app opriate, such as "Report C. rd on a Career Education Work- 
shop." ^ 

This cover page should contain, in addition to the title, the date of the report, the dait s o ^ e 
pp ct, the author of the report, and tlie notation that it is an evaluation report. If the r oort ; 
Sot a limited, specified audience, the names of those authorized to see it may be include^ m a s itt 
ment of the privacy of the information. 



11.2 



Recommendations. 



The recommendations generated as a resuit of 
the data analysis should be very acces^ibie. ! ' 
they are not resented on the first page, their 
location should be stated on the first page. 
Each recommendation should generaiiy be related to a specific decision faced by a reader or set of 
readers of the report, f n many ways the recommendations are the culmination of the study. The 
entire impact of the evaluation may rest on the clarity, relev ince, and credibility of the recommen- 
dations. In a technical report, each recommendation should generally be related to specific data. 
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U.3 IntrcxJuction, 



The introduction shouid provide e description 
of the project/activity evaluated and a brief 
overview of the purpose of the evaluation 
study and the intended audiences for the 
evaluation report. 



1 1 .4 Evaluation Procedures, 



This section should include descriptions of 
the instruments used, the overall evaluation 
framework, the limitations of the study, and 
the sources from which data were collected. 
If the report is technical, the procedures may appropriately be written in a style similar to an experi- 
mental iti dy. If the report is popular, the procedure should be briefly described in non technical 
language. 



11.5 Re^iuits. 



be reported in detail so that they can be related to 



The results should be presented both in simple 
summary tables when appropriate and in a 
short summary description. If the report is 
of a technical nature, the results may need to 
each recommendation. 
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CHECKLSST FOR RATING THE COMPONENT OF 
THE EVALUATION REPORT 



Instructions: Examine your {evaluation r^ort with respect to the following list of components 
by checking { ^ ) the appropriate responses. 

Needs 

Well Better Not Not 

Stated Statement Stated Applicable 



11.1 Cover page 



U 



□ 



□ 



□ 



1 1 .2 RecommerKJatioPj 



□ 



□ 



□ 



11.3 Introduction 



1 1 .3.1 Purpose of the evaluation 

1 1 .3.2 Questions to be answered 

1 1 .3.3 Audiences to be served 

1 1 .3.4 Document needs for the 
program 

1 1 .3.5 Subject matter 

1 1 .3.6 Objectives of program 

1 1 .3.7 1 nstructionat procedures/ 
program activities 

11.3.8 Participants: characteristics, 
type, and number involved 

1 1 .3.9 School setting and staff 
involvement 

11.3.10 Community setting and 
involvement 

1 1 .3 1 1 Costs: total and per pupil, 
amount needed for initial 
start up and for continuation 



□ 

n 

D 
G 

n 
□ 



□ 
□ 
□ 

□ 

n 
□ 

□ 

[J 



□ 
P 

D 
□ 

D 

n 
a 

□ 



□ 
□ 

□ 
n 
n 

□ 

□ 
□ 
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Needs 

Well Batter ■ . Not Not 
Stated Statement Stated, Applicable 

11.4 Evaluotion Procedures 



11.5 



11A1 


Overall framework 


a 


□ 




□ 

1 


11.4,2 


information sources {ooDutation 
sample) 


□ 


□ 


□ 




11.4.3 


Data collection Drocedurefi/rasfiarch 
design 


□ 






□ 


11.4.4 


Instruments used 


D 




n 


□ 


11.^.5 


Data analysis procedures 


D ■ 


□ 


D 


a 


11.4.6 


Limitations, constraint, and 
possible sources of bias 


□ 


n 


D 


n 


Results 












11.5.1 


Prooram outcomes in retationshin 
to needs of constituency 


□ 


n 


n 


□ 


11.5.2 


Attainment of objectives 


□ 


n 


1 — 1 


n 


11.53 


Unintended outcomes and social 
benefits 


□ 




n 


a 


11.5.4 


Cost-effectivbness 


□ 


a ■ 


□ 


□ 


11.5.5 


Value of the outcomes 


n 


a 


□ 


n 


11.5.6 


Conclusions 


n 




lZj 


D 


11.5.7 


Summary 


□ ■ 




□ 


□ 




\l CHECKLIST FOR RATING THE OVERALL QUALITY 
OF THE EVALUATION REPORT 



, instructions: Rate your evaluation r^ort according to the foHowing criteria by checking | ) the 
appropriate r^ponses. 

Yes No 

Accuracy 

□ n 11.1 Validity . Do^ the evaluation evidence present a true and fair picture of the 

program? 

□ □ 1 1 .2 Reliability. Would someone r^i^ting the evaluation obtain similar findings? 



a n 



1 1 .3 Appropriate analysis, is the eyeluation information ^aiyzed with appr(H3riate 

"techniques? [ / 



CI O 1 1 .4 Credible conclusions. Are the conclusions legitimately drawn from the 

findings? 

O □ 11.5 Objective reporting. Are the reajlts presentCKl and interprieted credibly, in 

the context in which they were collected, without overstatement? 

Utility 

□ O 1 1 .6 information a:ope. selection, and balance. Does the report focus on the most 

significant aspects of the program being evaluated? 

0 C] 11."/ Audience accommodation, is the report written with the audience's values, 

information needs, and bvel of knowledge in mind? 

fj 17] 1 "^-^i Timeliness . Is the report timed appropriately to facilitate use? 
Feasibility 

11 r] 1 1 .9 Reaiistic. Is the report constructive, specific, action-oriented, and sensitive 

to the resources avaiiable for program improvement? 

[ J [ J 11.10 Readibility. Is the language, length, and organization appropriate to reader's 

needs? | 

1 " j [1 11.11 Potiticai viability. Does the report illustrat^knowledge of and sensitivity to 

the political context in which the program operates? 
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12. Dis^minaticm^ What te€hnk|ui^ will be used to 
disidminai^ the evaluation findinp? 



The foHowing checklist iHustrates some techniques that can be used to communicate avalu 
ation findings to varied audiences. 



DISSEMINATION CHECKLIST 



Instructions: 
Yes No 



Survey the six techniques listed below and check the ones that would be most appro- 
priate for sharing your evaluation findings. 



□ 


D 


12,1 


Comprehensive technical report. 


□ 


□ 


12.2 


Technical report supplement with detailed recommendations for improve- 








ment for the program staff. 


□ 




12.3 


Executive summary. 


□ 


□ 


12.4 


Multiple short reports targeted to the information needs of different audiences. 


□ 


n 


12.5 


Popularized reports. 




n 


12.6 


News releases. 


(■ 1 


[□ 


12.7 


Oral presentation. ^' 




['.] 


12.8 


Audiovisual presentation. 



1?/^ Comprihensive Technical ReFK^*^. 



A technical report is a comprehensive descrip- 
tion of the evaluation procetlures^ the program 
itself, and the evaluation findings with conclud 
ing interpretations and recommendations. The 
technical report is intended as a back up or source document and should have limited istribution. 
It v/ould be appropriate for the funding sponsor and the key decision makers who are ^ely involved 
in the program. 




An appendix or supplement can be added to 
the copies of the technical report intended for 
the program staff. This supplement (for in- 
house review) would contain in-depth inior 
mation about specific aspects of the program that need to be improved. This supplement might also 
contain information that is somewhat sensitive or so highly derailed that it would be of interest only 
to the program staff. 
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12.3 Executive Summary, 



Most audiences receiving an c^tuation r^rt 
are more interested in the r^lts than in the 
methodology, tn addition, most audiences 
cannot afford the time to caret utty a 
well-documented and lengthy evaluation report. The executive summary is deigned to present a 
brief overview of the procedure, the important findings, and the general recommemlations. Obviously, 
ail the matei iai presented in the executive summary is contained in the technical report. This sumn;ary, 
usually one to ♦en pages in lengthy should make reference to the availability of the technical report. 
The reader of the summary can glean the important aspects of the study and, if interarted, can pursue 
more detailed information in the technical report. This technique keeps busy administrators informed 
and increases the potential for utilization of the evaluation data and r^ults. 



12,4 Multiple Short Reports . 



If there are more than one audience for the 
valuation r^ults, several rather short reports 
targeted for sp^ific audiences m^iy^ more 
feasible than a technical report ahd an execu- 
tive summary. Brickell (1974) found that decision makers prefer short (1 page) and medium (10 
pages) reports over longer reports. Brickell also found that most decision makers requeft short reports 
for themselves and longer on^ for their subordinates; however, when their subordinates were qiwied 
they requested short reports for thenselves and longer ones for their subordinates, and so on down 
the hierarchy. 
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12.5 Popularized Reports. 



A popularized report is a medium length (10- 



30 pages) report written without scphisticated 
terminology in a newsy, informative style. The 
popularized report provides a brief overview of 
the evaluation procedures and describes the program, especially unique or unusual activities, and the 
results of the evaluation. The results might be portrayed through anecdotes about the impact of 
the program on individual students as well as summaries of the major findings, conclusions, and 
recommendations. 



12.6 News Releases. 



Brief highlights of the evaluation finding^ can 
be communicated through local newspapers, 
newsletters, brochures, T.V. spots, and other 
media. An example of a news release is pre- 
sented in Figure 10 on page 89, 



12.7 Oral Presentation. 



If the audiences of the evaluation report arc 
not academicians or administrators, oral pres- 
entations may have more impact than written 
ones. Even if the audiences are academicians 
or administrators, a formal oral presentation snould be made. The oral presentation would include: 

• a <^.hort statement on the purposes of the evaliiation 

• highlights of evaluation techniques and results for each evaluation purpose 
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Fi^relO Sample Newt RttMW 

Shabazz ^Report Card' 
Good, With Exceptions 



By RICHAED HAWS 
Of The SUUp Journal Staff 



Mateolm Shabasz expenmenlal 
high school was fhund Monday oight 
to be *7ai¥}y effective overall in 
meetiog stU(}ent needi/' 

But in a report on tix^ Ihroe^year- 
old school presented to the School 
Bomrd. there were come '^^uestionfi 
as to the structure and direction of 
the program." 

The report, the work of a citizen 
committee of the East Educational 
Area Advisory Council, sppke of the 
''impression of disorganization." 



*Lack of Involvement* 

Most frequently mentioned m m 
weakne^ of the school was the "lack 
of involvement by some students.'" 



^Many parents feared tliat a large 
number of students were not attend- 
ing claisee and not participating in 
the ^gram/* the report aaid« 

Donald Hafeman, East Area at- 
tendance director and in charge of 
the icbool at 314 R Sherman Ave., 
praised the report for it4 objectivityt 
and Doled that it was the first citizen 
review of the coctroveraial ichodi. 

'Notable Featurea* 

The evaluators did praise the 
small class aise and r^ultant low 
teacher student ratios which it 
called ^notable features which con- 
tribute to effective learning.'* 



"The closenc^ of ieachei^ 
student relationships, facilitated 
by the school's small size, is a 



noaior cratiibutor to the penooal 
ana academic growth of atu^ntaf 
the report noted. 

But the evaiuatonr-lO East parents- 
rated the profc^onal itaff aa mora 
committed to the controvcmal achool 
than to the students. 

"^Probatdy becauM of their concern 
for those students who do not appMr 
to be involved at Shaban, the pamnta 
^d not rate student involvement and 
commitment as favorably aa Uiey 
rated the staff tnvcdvement," the 
IM^rants said 

A weaknMS in the prc^ram noted 
by parents, students, and teachers 
alike was the lack of adequate suu- 
plifi«, equipment and library fadiities. 

Ifafeman said the rf port would be 
carefidiy evaluated and used aa a 
guide toward making posaiUe im- 
provements in the s^ool. 



• recomrn€?ndations 

♦» 

• examples of strategies and techniques for implementing recommendations 

Presenting evaluation results to a group or to an indtviduoi provides an opportunity to question 
the results, discuss their implications, and Jr^termine what steps should be taken to use thei results. 
Through dialogue about the remits, a constructive plan of action should worked out to improve 
the program, make decisions, and/or yair^ increased support. 

A group presentation could use several speakers (e.g., a student, a teacher, an administrator, an 
evciluator, and a project director) to discuss the effectueness of the program from different points of 
view. Or various individuals could present the results :o constituents other than their own. For ex- 
ample, a teacher could present the results to a teachers' asscK:iation, a parent to the PTA, a student to 
the student body, and a superintendertt to the ^hool board. 




Evaluation results can be effectively communi- 
cated ^through audiovisual reports as weft as 
written and oral on^. Pictures, transparencies,^ 
slides, slide-tape presentations, films, and video 
it&pes can be used to orovide the audience .vith jn easy to-understand summary of the results. Audio- 
visual materials can afso be used to share highlights or key features of the program in an appealing 
manner. However, most decision makers prefer the print medtLiTi to the audiovisual (BrickeH 1974J. 
Audiovisual materials are mosf useful in group settings for formal presentations. 
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13. Uia. How will tlw evaluation findings be turned into action? 



To insure that the evaluation will improve the prc^ram being tested, you should develop an 
action plan for using the results of the evaluation. The following Use Checklist cites six points to 
consider when planning to use evaluation results. 

}/ USE CHECKLIST 

instructions: Survey the six questions on considerations about uses of your evaluation findings, and 
check ( / ) any appropriate r^}onses. 

/es No 

D □ 13.1 Data for decision makina. Will the evaluation reaults be used as a basis for 

making rational decisions about your program? 

□ D 13.2 Tvpe of decision. Are you clear about the type of decisions that the evaluation 

results should influence? 

□ n 13.3 Public Relations/Lobbying. Will the evaluation results be used to keep con- 

stituents informed of program outcome in order to gain support for the pro- 
gram or to influence future decisions and legislation regarding career educa- 
tion? 

□ □ 13.4 Useful recommendations. Are the evaluation rp^ommendations written in a 

manner that will facilitate their use? 

□ D 13.5 Personal barriers and facilitators. Have you examined the various personal 

reasons that individuals may have for adopting or resisting lite evaluation re- 
sults? 

D □ 13.6 Organizational barriers aid facilitators . Have y-ju examined the various or- 
ganizational barriers to end ways to facilitate use of the evaluation results? 



. 13.1 Data for decision making. Will the 
evaluation re^l^ be used as a basis for 
making more rational decisions ubout 
your program? 



Evaluation is not an end in itself. The results 
should be used as part of policy planning and 
management. A major purpose of valuation 
is to provide a basjs for making rational rather 
than intuitive decisions. Most decision makers 
/vouid like to have accurate information to 
help them make better decisions. Patton (1975) 



found that evaluation results are frequently used by decision makers but not often in the clear-cut 
and organization-shaking ways that social scientists sometimes believe these r^its should be used. 
Decision makers can use evaluation reajlts to: 
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resolve dtxibtt, confusions, and misunderstandings that different audiences may have 
about the program 

provide additional knowledge to support already known facts, to confirm obsecvationi, or 
to verify sitspicions 

supplement professional judgment 

fill in gaps with pieces of information about the program 

influence subsequent decisions on related programs 

provide documented evidence of program access and failures. 



Problems result not only when evaluation results are disregarded but also when the evaluation 
results are misused. Data is prevented from influencir.j decisions by numerous kinds of misuse, in- 
cluding the following three frequently found kinds: 



9. 



"Fuddlmg." An ar^ient and reyered skill of accomplished bureaucrats is "fuddling," the 
device of shuffiiing reports and recommendations by asking for signature 
and ciarificatloins without ever acuiaiiy implementing any of the recomn^ndations. 



"Misuse of Evaluation." Evaluation results may sometimes be used to justify conservative 
predecisions ratlier than progr^ive improvement of programs. 

"Overemphasis cn evaluation." On occasion, evaluation results can actually be overempha- 
sized. Muskin (1973) has documented cases in which decisions to terminate programs 
were based on evaluations that had unsound methods or faulty conceptualization. 



13.2 Type of decision. Are you clear 
about the type of decision that the 
evaluation results should influence? 



The decisions to be influenced by evaluatioir 
data include four types: planning decisions, 
structuring decisions, implementing decitions, 
and "recycling" deci^ons. Figure 1 1 illustrates 
th^ four tvpes of decisions and categorizes 
them by whether they are concerned with in- 
tended ends or means or actual ends or means. 

Intended ends are the objectives and planned outcomes of a program. Actual ends are the 

tual outcomes and reailts of a program. Intended means are the planned procedures for implementing 
a program. Actual means are the ways the program is actually Implemented. 
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Typ* of Daciiiom • 





Intended 


Actual 




Planning Decisions 


Recycling Decisions 


Ends 


To determine objectives 


To judge and react to 
attainment 




What needs should this 
program meet? 


Does tills program work 
■ or not? - 




Structuring Decisions 


implmnenting Decisions 


Means 


To design procedures 


To utilize, control and 
nf\m proc^ures 




How can these needs be 
met most effectively and 
efficiently? 


How can tfiis pro^rwn 
be improved? 



'Adapted from Stuff lebeam (1972). 



13.3 Public Relations/Lobbying. Will the 
evaluation nasufts be used for public rela- 
tions or lobbying? 



in addition to decision making, evaluation 
reiutts can be used for public relations or 
for lobbying: 



to keep constituents informed of the status and outcome of career education locally, 
statewide, or nationally 

f 

» to gain support for a particular program or for career education in general 

• to influence future decisions and legislation not directly related to the single program 
being evaluated 

Tl% results of an evaluation can be a defend of the status quo or a potent impetus for change. 
Listed below are some guidelines for shaping compelling evaluation results to lay before constituents 
such as parents, b( ards of education, advisory councils, and legislators. 

1 , lifustrate the outcomes of the program with examples of how individual students, teachers, 
and other participants benefitted from the prosram. 
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2. Compare the successes and weaknesses of your career-education program to those of 
other programs 

3. Stay abreast of the various political developments that influence your program. When 
possible. Illustrate how features of your program relate to 

— concerns expressed by legislatures or other governance bodies 

— emergmg federal, state; or local priorities 

— areas of inters to the key audiences for the evaluation 

, — new or pending legislation ' 

— new programs nnd projects 



13,4 Useful recommendations. Are 
the evaluation results and recommenda^ 
tions written in a manner that will 
enhance their significance and facilitate 
their use? 



Evaluation results and recommendations 
should contain those attr^tive elements of 
feature and format that will facilitate their 
use. Evaluation recommendations are more 
to be implemented when; 



lU^ely 



they are not overly time consuming to implement 
they are not overly costly to implement 
they are flexible and have alternatives 

they call for incremental changi! rather than total revamping of the program 

thi\ place the program staff in a central role {high profile) but do not make unreasonable 
demands on them 

they are short and concise yet complete 
they are easily communicated, not complex 



13.5 Personal barriers and facilitators . 
Have you examined the various personal 
reasons that individuals may have for 
adopting or resisting the evaluation 
results? 



There are often reasons, based both in fact and 
fancy, for resisting evaluations. In any group 
of individuals there will be considerable varia- 
tion in their willingness to adopt ev aluation 
recommendations. To facilitate the use of 
evaluation results, ou should identify those 
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individuals who will support and use evaluation results. These individuals can in turn help communi-^ 
cate the implications of the evaluation to those who are more resistant to change. It should be noted 
that some personalities are inherently resistant to using data to make decisions. In some cases, it 
will be impossible to change an intuitive decision maker into a rational one. 

Some traits that may differentiate users from nonusers of evaluation results are highlighted 
Deiow. 



Characteristics of a 
User of Evaluation Results 



ntakes rational decisions ba^ on 
logic and systematic iuffsrmation 



is influenced by group opinion 



takes risks and is op^n to chanq. 

is familiar with the uses of evaijation 

finds the evaluation approach 
consistent with personal values 

views a personal reward through the 
evaluation (e.g., gain in pow^r 
through having information that 
others don U 



Characteristics of a 
Non Userof Evaluation R^lts 



makes intuitive decisions based on 
personal re^lion, informal interpre- 
tation, and insight 

is more confident in own way of doing 
things than others' opinions 

is satisfied with the status quo 

is unaware 'hat evaEuations cm be useful 

finds the evaluation approach 
in conflict with personal values 

views a personal penalty through the 
evaluation {e.g., loss of job status) 



Personal Sa riers to Using Evaluation 

Both frontstaga and backstage concerns about the evatuation sliould be considered. Because 
evaluation is often threatening, irrational as well as rational resistance may surface. Some of the 
reasons why a resister opposes evaluations are highligh ted below (adapted from Davis and Salasin, 
1975h 

• views evaluation as somewhat unclear or confusing 

• sees limited personal help or benefits from the evaination 

• sees litti * of value to be gained by the project from the evaluation 
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• views evaluation ^proach as a poor fit with the program 

• views the evaluation approach as a poor fit with the style and values of the Midience 

• sees the evaluation as invalid in basic concept and methods 

• , foresees native consequences of the evaluation 

« feels it is a poor or in^propriate time to be evaluated (e.g., too ^oon, too late, too 
unimportant, etc.) 



Reducing Barriers . 

Despite the many reasons for reluctance to use the reaitts of an evaiuatton, there are ^ number 
of possible ways to reduce the resistance, and eleven of these are listed below: 

• Appraise resistances deemed to be rational and consider restating the evaluation results 
in a more diplomatic manner 

• Use advisory groups who represent opposing factions 

• Approach implementation of the evaluation slowly; intrcniuce new eiem^ts a few at a 
time, begi|ining with the least disturbing 

• Compromise and st^ttle on partial change if necessary 

• Use listening as a deliberate approach; allow persons to express their resistar^e 

• Employ . ?tiona{ persuasion 

• Use selective individual counseling for those who have unique problems or strong resistance 

• Resolve complex resistance thi ough group dynamics (e.g., values clarification, T«^oups, 
open discussion at retread, etc.) 

• Ptan, direct, tangible rewards for persons instrumental to using the evatuaticKi rtsuits 

• Communicate or share the benefits from implementing the evaluation results wtth persons 
concern^ 

• Seek and use direct and personal support from the top of the organization 

I Forces for and against use of evaluation 
Qragpization^l barriers and ^^^^^^^ often be numerous. In addi^ 
fgctlitators. Have you exammed the tion to the persona! bisrriers and facili^ 
various organizational barriers to and ways tators outlined above, there can also be^ 
to facilitate use of the evaluation results? depending on the health of the organization- 
I organizational barriers and faciiitatofi. This 
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section deicrib«s faciiitators, ^id it provictei guidofiras for ovtrcoming tilt bvHart and strength- 
sning tiie facilitators. ^ 

Offlanizationat Faciiitatofs 

Some organizations are healthier than others and tend to be more responsive to evaluation re- 
sults. Some characteristics of a healthy organization, one that is rsc^tive to evaluation, era outlined 
below (adapted from Davis and Sala^n, 1875): 

Organizational goals are clear and written down. 

Administi^tion is aipportlye. 1 

Communication is open. 

Colleagues enjoy mutual sjppot 

Participation in decision making is widesjpread. 

Morale is high. 

Adequate time is allowed for reflection and for t^ing new Ideas, 
Power is decentralized and distribute. 
Organization is reasonably affluent and secure. 
Constiments are supportf'%. 
There is a history of successful innovations. 
The chief decision maker has a reasonably short tenure. 
Staff members are rewarded fc^r performance rather than status. 
The highest ranking official in ^e organization is a self-renewing, goal-oriented person. 

Strengthening Organizational Facilitatdrs 

i 

Evaluation results will meet greater support in healthy orpnizations. Therefore, it is advanta- 
geous to improve an organization's hea\tn In as many of ^e above areas ^ possible. Some steps 
for strengdrening organizational fa^liitstors are: ^ 

• advcK^ating self-renewal of staff and employees 

• encouraging experimentation and the right to make mistakes 
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ei^pun^ing attendance at maetlngs, diicusslons, "brown baggers." etc. 

s 

arranging nwards and recognition related to performance 

Informing corKerned indlvtduali of the progresi of the evaluation and periodically ajking 
for their feedback 



Summary 

Systematic stepi should be planned for facilitating use of the evaluation results and recommenda- 
tions. The process of planning for use of evaluation shwild include: 

• building a commitment to draw upon evaluation results for making rational datii-baaed 
decisions 

• decldingwhrttypesofdeckoi^ and what' specific decisions should be influenced by the 
evaluation results 

• planning broad uses of the evaluation results, such as upgrading public relations, lol^ying, 
influencing legislation, and influencing futijre program decisions 

• designing the evaluation results and recommendations to be as compelling and useable as 
po^bte ' • 

• developing sensitivity to the expectations and resistam^s of Individuals who are expected 
to use the evaluation results^ their needs may be accommodated 

• evaluating the organizational setting for the evaluation so barriers can be overcome and 
strengths reinforced 
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CURRENT COMPETENCY 




EVALUATJON QUESTIONS 
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1. 


t 
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needed? 
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o 


o. 
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2. 


Audi6n£te Who shDufd the fivatuation 
serve? 


O 


o 


o 


O 


0 


3. 


Questions What soocific ouestions shouid 
the evaluatron aniwer ? 


0 


o 


o 


O 


0 


4. 


Process. How will the evaluation be 
accomplished? 


0 


o 


o 


O 


0 


5. 


Staff. What tvoe of evaluation servioM if 
any, should be us^? 


0 


o 


o 


O 


0 


6. 


ResDonslbiiitv/Authoritv. What are the 
respective roles and rdsponsibflitiis of 
program staff and evaluatort? 
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o 


o 


O 


0 




Unioueness. What uniaue features of career 
education influence its valuation? 
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8. 


Piannina Standards. What are the character- 
istics of a "good" evaluation plan? 
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o 
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^. 


Instrument Standards. What are the charac- 

te-istics of a "good" evaluation t rratrument? 
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o 


o 


o 


0 


10. 


Politics. What oolitical factors miaht affect 
use of the evaluation results? 
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n. 


Report Standards. What are tiie characteristics 
of a "good" evaluation report? 
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12. 


Dissemination. What techniau^ will be used 
to disemlnate the findings? 


0 

None 
0 


o 
1 


o o 

^ 

Moderate 
2 3 


o 

High 
4 


13. 


Use. How will the evaluation finding be 
turned into action? 
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Compare your pretest profile from pa^ 5 of the handbook with your postteit prefilt. Enter 
your pretest ratings on the posttest profile usln^ a different color pen. Connect the dretes to form 
your pre- and post profile as illustrated below. 



None 




Moderate 


High 






0 ' 
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3 4 
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Purpose 
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2. 


Audience 
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3. 


Questions 
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4. 
• 


Process 
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o o 


• 

13. 


> 

n Use 



Pretest 



Postt^t 



Calculate the amount of change in your knowledge of each of the thirteen tppTSN Don't be 
surprised if you find some ratings lower on the post-test than the pretest. An increased awareness 
of the complexity of the issues involved in valuation may serve to increase your perc^tion of hqw 
much more there is to learn. The comparison of profiles is primarily useful to help you identify 
knowledge areas that are still below your expectations and areas that you' now feel fairly comfor- 
table performing. Yau may want to review units or use additional resource to learn more about 
topics that are still below your expectations. 

This handbook was designed to stimulate ycHir awareness of the many tsues, activities and hoctd-' 
aches that go into making an evaluation top-notch. Many of these isai^ are easiffi- to tiik about than 
to actually integrate into daily profe^jsionai activities. In other words, it is much easier to analyze 
what needs to be done .than to implement ideas in a world of constraints, changing signals, interper- 
sonal conflicts, and heavy workloads. This handbook is not meant to overwhelm you with all die 
issues to be considered In an evaluation. Select a few new ideas that appeal to you arul try to act on 
them in your next evaluation. Remember, changes are more frequent and lasting when they are in- 
cremental (small steps), uncomplicated, and flexible. Begin to think of situation^where you can 
apply your new knowledge. , 

This handbook is a compr#iensive sourcebook about the key ingredients of an evaluation. But, 
because the handbook provides breadth of information it gives only a taste or survey of many ideas 
-rather than in-depth knowledge and detailed procedures. Many other reference books, »ticles, and 
reports are cited throughout the handbook. You are encouraged to,,turn to the^ materials to Investi- 
gate topics of inters in more detail. 
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